Merge branch 'master' into investigation

Conflicts:
	org-babel.org

org-babel.org

@@ -208,174 +208,6 @@ would then be [[#sandbox][the sandbox]].
 
 
 * Tasks [35/57]
-** PROPOSED implement fully recursive evaluation machinery
-   The current parser / evaluator fails with greater levels of nested
-   function block calls (example below). If we want to overcome this I
-   think we'd have to redesign some of the evaluation
-   mechanism. Seeing as we are also facing issues like dealing with
-   default argument values, and seeing as we now know how we want the
-   library of babel to behave in addition to the source blocks, now
-   might be a good time to think about this. It would be nice to do
-   the full thing at some point, but otoh we may not consider it a
-   massive priority.
-   
-   AIui, there are two stages: (i) construct a parse tree, and (ii)
-   evaluate it and return the value at the root. In the parse tree
-   each node represents an unevaluated value (either a literal value
-   or a reference). Node v may have descendent nodes, which represent
-   values upon which node v's evaluation depends. Once that tree is
-   constructed, then we evaluate the nodes from the tips towards the
-   root (a post-order traversal).
-
-   [This would also provide a solution for concatenating the STDOUTs
-   of called blocks, which is a [[*allow%20output%20mode%20to%20return%20stdout%20as%20value][task below]]; we concatenate them in
-   whatever order the traversal is done in.]
-
-   In addition to the variable references (i.e. daughter nodes), each
-   node would contain the information needed to evaluate that node
-   (e.g. lang body). Then we would pass a function postorder over the
-   tree which would call o-b-execute-src-block at each node, finally
-   returning the value at the root.
-
-   Fwiw I made a very tentative small start at stubbing this out in
-   org-babel-call.el in the 'evaluation' branch. And I've made a start
-   at sketching a parsing algorithm below.
-
-*** discussion
-I believe that this issue should be addressed as a bug rather than as
-a point for new development.   The code in [[file:lisp/org-babel-ref.el][org-babel-ref.el]] already
-resolves variable references in a recursive manner which *should* work
-in the same manner regardless of the depth of the number of nested
-function calls.  This recursive evaluation has the effect of
-implicitly constructing the parse tree that your are thinking of
-constructing explicitly.
-
-Through using some of the commented out debugging statements in
-[[file:lisp/org-babel-ref.el][org-babel-ref.el]] I have looked at what may be going wrong in the
-current evaluation setup, and it seems that nested variables are being
-set using the =:var= header argument, and these variables are being
-overridden by the *default* variables which are being entered through
-the new functional syntax (see the demonstration header below).
-
-I believe that once this bug is fixed we should be back to fully
-resolution of nested arguments.  We should capture this functionality
-in a test to ensure that we continue to test it as we move forward.  I
-can take a look at implementing this once I get a chance.
-
-**** demonstration
-After uncommenting the debugging statements located [[file:lisp/org-babel-ref.el::message%20format%20first%20second%20S%20S%20new%20refere%20new%20referent%20debugging][here]] and more
-importantly [[file:lisp/org-babel-ref.el::message%20nested%20args%20S%20args%20debugging][here]], we can see that the current reference code does
-evaluate the references correctly, and it uses the =:var= header
-argument to set =a=8=, however the default variables specified using
-the functional syntax in =adder(a=3, b=2)= is overriding this
-specification.
-
-***** doesn't work with functional syntax
-
-#+srcname: adder-func(a=3, b=2)
-#+begin_src python 
-a + b
-#+end_src
-
-#+resname: adder-func
-: 5
-
-#+srcname: after-adder-func(arg=adder-func(a=8))
-#+begin_src python 
-arg
-#+end_src
-
-#+resname: after-adder-func
-: 5
-
-***** still does work with =:var= syntax
-
-so it looks like regardless of the syntax used we're not overriding
-the default argument values.
-
-#+srcname: adder-header
-#+begin_src python :var a=3 :var b=2
-a + b
-#+end_src
-
-#+resname: adder-header
-: 5
-
-#+srcname: after-adder-header
-#+begin_src python :var arg=adder-header(a=8, b=0)
-arg
-#+end_src
-
-#+resname: after-adder-header
-: 5
-
-
-*** Parse tree algorithm
-    Seeing as we're just trying to parse a string like
-    f(a=1,b=g(c=2,d=3)) it shouldn't be too hard. But of course there
-    are 'proper' parsers written in elisp out there,
-    e.g. [[http://cedet.sourceforge.net/semantic.shtml][Semantic]]. Perhaps we can find what we need -- our syntax is
-    pretty much the same as python and R isn't it?
-
-    Or, a complete hack, but maybe it would be we easy to transform it
-    to XML and then parse that with some existing tool?
-    
-    But if we're doing it ourselves, something very vaguely like this?
-    (I'm sure there're lots of problems with this)
-
-#+srcname: org-babel-call-parse(call)
-#+begin_src python 
-  ## we are currently reading a reference name: the name of the root function
-  whereami = "refname"
-  node = root = Node()
-  for c in call_string:
-      if c == '(':
-          varnum = 0
-          whereami = "varname" # now we're reading a variable name
-      if c == '=':
-          new = Node()
-          node.daughters = [node.daughters, new]
-          new.parent = node
-          node = new
-          whereami = "refname"
-      if c == ',':
-          whereami = "varname"
-          varnum += 1
-      elif c == ')':
-          node = node.parent
-      elif c == ' ':
-          pass
-      else:
-          if whereami = "varname":
-              node.varnames[varnum] += c
-          elif whereami = "refname":
-              node.name += c
-#+end_src
-    
-*** Example that fails
-    
-#+srcname: adder(a=0, b=0)
-#+begin_src python 
-a+b
-#+end_src
-
-#+srcname: one()
-#+begin_src python :results silent
-1
-#+end_src
-
-This works
-#+srcname: level-one-nesting
-#+begin_src python :var arg=adder(a=one(),b=one())
-arg
-#+end_src
-
-But this doesn't
-#+srcname: level-one-nesting
-#+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=one()))
-arg
-#+end_src
-
 ** PROPOSED Default args
    This would be good thing to address soon. I'm imagining that
    e.g. here, the 'caller' block would return the answer 30. I believe
@@ -2449,6 +2281,259 @@ plot data using 1:2 with lines
 
 
 * Bugs [24/36]
+** TODO Fix nested evaluation
+   The current parser / evaluator fails with greater levels of nested
+   function block calls (example below).
+
+*** Initial statement [ded]
+    If we want to overcome this I think we'd have to redesign some of
+    the evaluation mechanism. Seeing as we are also facing issues like
+    dealing with default argument values, and seeing as we now know
+    how we want the library of babel to behave in addition to the
+    source blocks, now might be a good time to think about this. It
+    would be nice to do the full thing at some point, but otoh we may
+    not consider it a massive priority.
+    
+    AIui, there are two stages: (i) construct a parse tree, and (ii)
+    evaluate it and return the value at the root. In the parse tree
+    each node represents an unevaluated value (either a literal value
+    or a reference). Node v may have descendent nodes, which represent
+    values upon which node v's evaluation depends. Once that tree is
+    constructed, then we evaluate the nodes from the tips towards the
+    root (a post-order traversal).
+
+    [This would also provide a solution for concatenating the STDOUTs
+    of called blocks, which is a [[*allow%20output%20mode%20to%20return%20stdout%20as%20value][task below]]; we concatenate them in
+    whatever order the traversal is done in.]
+
+    In addition to the variable references (i.e. daughter nodes), each
+    node would contain the information needed to evaluate that node
+    (e.g. lang body). Then we would pass a function postorder over the
+    tree which would call o-b-execute-src-block at each node, finally
+    returning the value at the root.
+
+    Fwiw I made a very tentative small start at stubbing this out in
+    org-babel-call.el in the 'evaluation' branch. And I've made a start
+    at sketching a parsing algorithm below.
+**** Parse tree algorithm
+    Seeing as we're just trying to parse a string like
+    f(a=1,b=g(c=2,d=3)) it shouldn't be too hard. But of course there
+    are 'proper' parsers written in elisp out there,
+    e.g. [[http://cedet.sourceforge.net/semantic.shtml][Semantic]]. Perhaps we can find what we need -- our syntax is
+    pretty much the same as python and R isn't it?
+
+    Or, a complete hack, but maybe it would be we easy to transform it
+    to XML and then parse that with some existing tool?
+    
+    But if we're doing it ourselves, something very vaguely like this?
+    (I'm sure there're lots of problems with this)
+
+#+srcname: org-babel-call-parse(call)
+#+begin_src python 
+  ## we are currently reading a reference name: the name of the root function
+  whereami = "refname"
+  node = root = Node()
+  for c in call_string:
+      if c == '(':
+          varnum = 0
+          whereami = "varname" # now we're reading a variable name
+      if c == '=':
+          new = Node()
+          node.daughters = [node.daughters, new]
+          new.parent = node
+          node = new
+          whereami = "refname"
+      if c == ',':
+          whereami = "varname"
+          varnum += 1
+      elif c == ')':
+          node = node.parent
+      elif c == ' ':
+          pass
+      else:
+          if whereami = "varname":
+              node.varnames[varnum] += c
+          elif whereami = "refname":
+              node.name += c
+#+end_src
+    
+*** discussion
+I believe that this issue should be addressed as a bug rather than as
+a point for new development.   The code in [[file:lisp/org-babel-ref.el][org-babel-ref.el]] already
+resolves variable references in a recursive manner which *should* work
+in the same manner regardless of the depth of the number of nested
+function calls.  This recursive evaluation has the effect of
+implicitly constructing the parse tree that your are thinking of
+constructing explicitly.
+
+Through using some of the commented out debugging statements in
+[[file:lisp/org-babel-ref.el][org-babel-ref.el]] I have looked at what may be going wrong in the
+current evaluation setup, and it seems that nested variables are being
+set using the =:var= header argument, and these variables are being
+overridden by the *default* variables which are being entered through
+the new functional syntax (see the demonstration header below).
+
+I believe that once this bug is fixed we should be back to fully
+resolution of nested arguments.  We should capture this functionality
+in a test to ensure that we continue to test it as we move forward.  I
+can take a look at implementing this once I get a chance.
+
+**** demonstration
+After uncommenting the debugging statements located [[file:lisp/org-babel-ref.el::message%20format%20first%20second%20S%20S%20new%20refere%20new%20referent%20debugging][here]] and more
+importantly [[file:lisp/org-babel-ref.el::message%20nested%20args%20S%20args%20debugging][here]], we can see that the current reference code does
+evaluate the references correctly, and it uses the =:var= header
+argument to set =a=8=, however the default variables specified using
+the functional syntax in =adder(a=3, b=2)= is overriding this
+specification.
+
+***** doesn't work with functional syntax
+
+#+srcname: adder-func(a=3, b=2)
+#+begin_src python 
+a + b
+#+end_src
+
+#+resname: adder-func
+: 5
+
+#+srcname: after-adder-func(arg=adder-func(a=8))
+#+begin_src python 
+arg
+#+end_src
+
+#+resname: after-adder-func
+: 5
+
+***** still does work with =:var= syntax
+
+so it looks like regardless of the syntax used we're not overriding
+the default argument values.
+
+#+srcname: adder-header
+#+begin_src python :var a=3 :var b=2
+a + b
+#+end_src
+
+#+resname: adder-header
+: 5
+
+#+srcname: after-adder-header
+#+begin_src python :var arg=adder-header(a=8, b=0)
+arg
+#+end_src
+
+#+resname: after-adder-header
+: 5
+
+*** Set of test cases
+**** Both defaults provided in definition
+#+srcname: adder1(a=10,b=20)
+#+begin_src python
+a+b
+#+end_src
+
+#+resname: adder1
+: 30
+
+****** DONE Rely on defaults
+#+lob: adder1( )
+
+#+resname: adder1( )
+: 30
+
+## should be 30
+## OK, but
+******* TODO empty parens () not recognised as lob call
+	E.g. remove spaces between parens above
+
+****** TODO One supplied, one default
+#+lob: adder1(a=0)
+
+#+resname: adder1(a=0)
+: 30
+## should be 10
+
+#+lob: adder1(b=0)
+
+#+resname: adder1(b=0)
+: 30
+## should be 10
+
+****** TODO Both supplied
+#+lob: adder1(a=1,b=2)
+
+#+resname: adder1(a=1,b=2)
+: 30
+
+## should be 3
+
+**** One arg lacks default in definition
+#+srcname: adder2(a=10,b)
+#+begin_src python
+a+b
+#+end_src
+****** TODO Rely on defaults (one of which is missing)
+#+lob: adder2( )
+
+[no output]
+
+## should be error: b has no default
+
+****** TODO Default over-ridden
+#+lob: adder2(a=1)
+
+[no output ]
+## should be error: b has no default
+
+****** DONE Missing default supplied
+#+lob: adder2(b=1)
+
+#+resname: adder2(b=1)
+: 11
+
+## should be 11
+## OK
+
+****** DONE One over-ridden, one supplied
+#+lob: adder2(a=1,b=2)
+
+#+resname: adder2(a=1,b=2)
+: 3
+## should be 3
+
+*** Example that fails
+    
+#+srcname: adder(a=0, b=99)
+#+begin_src python 
+a+b
+#+end_src
+
+#+resname: adder
+: 99
+
+
+#+srcname: one()
+#+begin_src python :results silent
+2
+#+end_src
+
+
+#+srcname: level-one-nesting
+#+begin_src python :var arg=adder(a=one(),b=one())
+arg
+#+end_src
+
+#+resname:
+: 99
+
+#+srcname: level-one-nesting
+#+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=one()))
+arg
+#+end_src
+
+#+resname:
+: 99
+
 ** TODO allow srcname to omit function call parentheses
    Someone needs to revisit those regexps. Is there an argument for
    moving some of the regexps used to match function calls into