pulled org-babel.org from session branch

org-babel.org

@@ -1,7 +1,7 @@
 #+OPTIONS:    H:3 num:nil toc:t
 #+TITLE: org-babel --- facilitating communication between programming languages and people
 #+SEQ_TODO:  TODO PROPOSED | DONE DEFERRED REJECTED
-#+STARTUP: oddeven
+#+STARTUP: oddeven hideblocks
 
 * Introduction
 
@@ -114,8 +114,9 @@ table, allowing the test suite to be run be evaluation of the table
 and the results to be collected in the same table.
 
 
-* Tasks [22/37]
-** TODO Create objects in top level (global) environment in R? [0/5]
+* Tasks [22/38]
+** TODO Create objects in top level (global) environment [0/5]
+*sessions*
 
 *** initial requirement statement [DED]
    At the moment, objects created by computations performed in the
@@ -189,6 +190,222 @@ or if you think any parts might be confusing for people coming from
 Sweave.  I'll hopefully find some time to work on this later in the
 week.
 
+*** can functional and interpreted/interactive models coexist?
+
+Even though both of these use the same =*R*= buffer the value of =a=
+is not preserved because it is assigned inside of a functional
+wrapper.
+
+#+srcname: task-R-sessions
+#+begin_src R 
+a <- 9
+b <- 21
+a + b
+#+end_src
+
+#+srcname: task-R-same-session
+#+begin_src R 
+a
+#+end_src
+
+This functional wrapper was implemented in order to efficiently return
+the results of the execution of the entire source code block.  However
+it inhibits the evaluation of source code blocks in the top level,
+which would allow for persistence of variable assignment across
+evaluations.  How can we allow *both* evaluation in the top level, and
+efficient capture of the return value of an entire source code block
+in a language independent manner?
+
+Possible solutions...
+1) we can't so we will have to implement two types of evaluation
+   depending on which is appropriate (functional or imperative)
+2) we remove the functional wrapper and parse the source code block
+   into it's top level statements (most often but not always on line
+   breaks) so that we can isolate the final segment which is our
+   return value.
+3) we add some sort of "#+return" line to the code block
+4) we take advantage of each languages support for meta-programming
+   through =eval= type functions, and use said to evaluate the entire
+   blocks in such a way that their environment can be combined with the
+   global environment, and their results are still captured.
+5) I believe that most modern languages which support interactive
+   sessions have support for a =last_result= type function, which
+   returns the result of the last input without re-calculation.  If
+   widely enough present this would be the ideal solution to a
+   combination of functional and imperative styles.
+
+None of these solutions seem very desirable, but for now I don't see
+what else would be possible.
+
+Of these options I was leaning towards (1) and (4) but now believe
+that if it is possible option (5) will be ideal.
+
+**** (1) both functional and imperative evaluation
+Pros
+- can take advantage of built in functions for sending regions to the
+  inferior process
+- retains the proven tested and working functional wrappers
+
+Cons
+- introduces the complication of keeping track of which type of
+  evaluation is best suited to a particular context
+- the current functional wrappers may require some changes in order to
+  include the existing global context
+
+**** (4) exploit language meta-programming constructs to explicitly evaluate code
+Pros
+- only one type of evaluation
+
+Cons
+- some languages may not have sufficient meta-programming constructs
+
+**** (5) exploit some =last_value= functionality if present
+
+Need to ensure that most languages have such a function, those without
+will simply have to implement their own similar solution...
+
+| language   | =last_value= function       |
+|------------+-----------------------------|
+| R          | .Last.value                 |
+| ruby       | _                           |
+| python     | _                           |
+| shell      | see [[* last command for shells][last command for shells]] |
+| emacs-lisp | see [[* emacs-lisp will be a special case][special-case]]            |
+
+#+srcname: task-last-value
+#+begin_src ruby
+82 + 18
+#+end_src
+
+***** last command for shells
+Do this using the =tee= shell command, and continually pipe the output
+to a file.
+
+Got this idea from the following [[http://linux.derkeiler.com/Mailing-Lists/Fedora/2004-01/0898.html][email-thread]].
+
+suggested from mailing list
+
+#+srcname: bash-save-last-output-to-file
+#+begin_src sh 
+while read line 
+do 
+  bash -c "$line" | tee /tmp/last.out1 
+  mv /tmp/last.out1 /tmp/last.out 
+done
+#+end_src
+
+another proposed solution from the above thread
+
+#+srcname: bash-save-in-variable
+#+begin_src sh 
+#!/bin/bash 
+# so - Save Output. Saves output of command in OUT shell variable. 
+OUT=`$*` 
+echo $OUT 
+#+end_src
+
+and another
+
+#+begin_quote
+.inputrc: 
+"^[k": accept-line 
+"^M": " | tee /tmp/h_lastcmd.out ^[k" 
+
+.bash_profile: 
+export __=/tmp/h_lastcmd.out 
+
+If you try it, Alt-k will stand for the old Enter; use "command $__" to 
+access the last output. 
+
+Best, 
+
+--
+
+Herculano de Lima Einloft Neto
+#+end_quote
+
+***** emacs-lisp will be a special case
+While it is possible for emacs-lisp to be run in a console type
+environment (see the =elim= function) it is *not* possible to run
+emacs-lisp in a different *session*.  Meaning any variable set top
+level of the console environment will be set *everywhere* inside
+emacs.  For this reason I think that it doesn't make any sense to
+worry about session support for emacs-lisp.
+
+*** Further thoughts on 'scripting' vs. functional approaches
+
+    These are just thoughts, I don't know how sure I am about this.
+    And again, perhaps I'm not saying anything very radical, just that
+    it would be nice to have some options supporting things like
+    receiving text output in the org buffer.
+
+    I can see that you've already gone some way down the road towards
+    the 'last value' approach, so sorry if my comments come rather
+    late. I am concerned that we are not giving sufficient attention
+    to stdout / the text that is returned by the interpreters. In
+    contrast, many of our potential users will be accustomed to a
+    'scripting' approach, where they are outputting text at various
+    points in the code block, not just at the end. I am leaning
+    towards thinking that we should have 2 modes of evaluation:
+    'script' mode, and 'functional' mode.
+
+    In script mode, evaluation of a code block would result in *all*
+    text output from that code block appearing as output in the org
+    buffer, presumably as an #+begin_example...#+end_example. There
+    could be an :echo option controlling whether the input commands
+    also appear in the output. [This is like Sweave].
+
+    In functional mode, the *result* of the code block is available as
+    an elisp object, and may appear in the org buffer as an org
+    table/string, via the mechanisms you have developed already.
+
+    One thing I'm wondering about is whether, in script mode, there
+    simply should not be a return value. Perhaps this is not so
+    different from what exists: script mode would be new, and what
+    exists currently would be functional mode.
+
+    I think it's likely that, while code evaluation will be exciting
+    to people, a large majority of our users in a large majority of
+    their usage will not attempt to actually use the return value from
+    a source code block in any meaningful way. In that case, it seems
+    rather restrictive to only allow them to see output from the end
+    of the code block.
+
+    Instead I think the most accessible way to introduce org-babel to
+    people, at least while they are learning it, is as an immensely
+    powerful environment in which to embed their 'scripts', which now
+    also allows them to 'run' their 'scripts'. Especially as such
+    people are likely to be the least capable of the user-base, a
+    possible design-rule would be to make the scripting style of usage
+    easy (default?), perhaps requiring a special option to enable a
+    functional style. Those who will use the functional style won't
+    have a problem understanding what's going on, whereas the 'skript
+    kiddies' might not even know the syntax for defining a function in
+    their language of choice. And of course we can allow the user to
+    set a variable in their .emacs controlling the preference, so that
+    functional users are not inconveniennced by having to provide
+    header args the whole time.
+
+    Please don't get the impression that I am down-valuing the
+    functional style of org-babel. I am constantly horrified at the
+    messy 'scripts' that my colleagues produce in perl or R or
+    whatever! Nevertheless that seems to be how a lot of people work.
+    
+    I think you were leaning towards the last-value approach because
+    it offered the possibility of unified code supporting both the
+    single evaluation environment and the functional style. If you
+    agree with any of the above then perhaps it will impact upon this
+    and mean that the code in the two branches has to differ a bit. In
+    that case, functional mode could perhaps after all evaluate each
+    code block in its own environment, thus (re)approaching 'true'
+    functional programming (side-effects are hard to achieve).
+
+#+begin_src sh
+ls > files
+echo "There are `wc -l files` files in this directory"
+
+#+end_src
+
 *** TODO rework all source codes to use inferior-processes-buffers
 
 this will involve...
@@ -198,6 +415,27 @@ this will involve...
 3) functions for retrieving results from the *sessions* buffers which
    can be overridden by each source code
 
+**** DONE R
+
+#+srcname: task-R-with-inf-process-buffer
+#+begin_src R 
+a <- 8
+b <- 9
+c <- 10
+a + b
+#+end_src
+
+**** TODO Ruby
+
+#+srcname: ruby-use-last-output
+#+begin_src ruby 
+a = 1
+b = 2
+c = 3
+(a + b) * c
+#+end_src
+
+
 *** TODO implement a *session* header argument
 
 use this header argument to override the default *session* buffer
@@ -284,7 +522,6 @@ org-mode core
 This should be implemented in the org-mode core
      
 
-
 *** DEFERRED send code to inferior process
 Another thought on this topic: I think we will want users to send
 chunks of code to the interpreter from within the *Org Edit Src*
@@ -347,6 +584,11 @@ waiting for guidance from those more familiar with yasnippets
    - [[file:lisp/org-babel-ref.el::TODO%20allow%20searching%20for%20names%20in%20other%20buffers][org-babel-ref.el:searching-in-other-buffers]]
    - [[file:lisp/org-babel.el::defun%20org-babel%20find%20named%20result%20name][org-babel.el#org-babel-find-named-result]]
 
+** TODO figure out how to handle errors during evaluation
+   R has a try function, with error handling, along the lines of
+   python. I bet ruby does too. Maybe more of an issue for functional
+   style; in my proposed scripting style the error just gets dumped to
+   the org buffer and the user is thus alerted.
 ** TODO figure out how to handle graphic output
 This is listed under [[* graphical output][graphical output]] in out objectives.
 
@@ -545,7 +787,6 @@ msg + "_y_python"
 (concat msg "_elisp")
 #+end_src
 
-
 ** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format
    I haven't thought about this properly. Just noting it down. What
    Sweave uses is called "R noweb" (.Rnw).