Tables: Support for references to remote tables

Formulas can now refer to fields and regions in other tables, be it in
the same file or even in different files.

This also opens the door for more interesting R support, because
arbitrary tables in a file can now be accessed with appropriate
commands.

Finally, this commit also introduces a convenience for creating
references in tables.  For the purpose of references, it will always
be *assumed* that there is another hline at the end if each table.  So
for example in a table that has only a single hline, to separate
header from data, you can now access all of column two with
"@I$2..@II$2".

ORGWEBPAGE/Changes.org

@@ -48,6 +48,23 @@
 	   into normal lines. 
 	 - if it is a normal line, turn all lines into headlines.
 
+*** Table formulas may now reference fields in other tables
+
+
+You may also reference constants, fields and ranges from a different table,
+either in the current file or even in a different file.  The syntax is
+
+: remote(NAME-OR-ID,REF)
+
+where /NAME/ can be the name of a table in the current file as
+set by a =#+TBLNAME: NAME= line before the table.  It can also be
+the ID of an entry, even in a different file, and the reference
+then refers to the first table in that entry.  /REF/ is an
+absolute field or range reference, valid in the referenced
+table.  Note that since there is no "current filed" for the
+cursor, all row and column references must be absolute, not
+relative.
+
 * Version 6.18
 ** Incompatible changes
 

doc/ChangeLog

@@ -1,3 +1,7 @@
+2009-01-24  Carsten Dominik  <carsten.dominik@gmail.com>
+
+	* org.texi (References): Add information about remote references.
+
 2009-01-21  Carsten Dominik  <carsten.dominik@gmail.com>
 
 	* orgcard.tex: Document `C-c RET' in tables.

doc/org.texi

@@ -1990,6 +1990,27 @@ lines.  These are described below, see @ref{Advanced features}.  All
 names must start with a letter, and further consist of letters and
 numbers.
 
+@subsubheading Remote references
+@cindex remote references
+@cindex references, remote
+@cindex references, to a different table
+@cindex name, of column or field
+@cindex constants, in calculations
+
+You may also reference constants, fields and ranges from a different table,
+either in the current file or even in a different file.  The syntax is
+
+@example
+remote(NAME-OR-ID,REF)
+@end example
+
+@noindent
+where NAME can be the name of a table in the current file as set by a
+@code{#+TBLNAME: NAME} line before the table.  It can also be the ID of an
+entry, even in a different file, and the reference then refers to the first
+table in that entry.  REF is an absolute field or range reference as
+described above, valid in the referenced table.
+
 @node Formula syntax for Calc, Formula syntax for Lisp, References, The spreadsheet
 @subsection Formula syntax for Calc
 @cindex formula syntax, Calc

lisp/ChangeLog

@@ -1,3 +1,19 @@
+2009-01-24  Carsten Dominik  <carsten.dominik@gmail.com>
+
+	* org-table.el (org-table-get-specials): Add an imagined hline at
+	the end of the table.  This can be useful for references that want
+	to go to the end of the table.  Also fix bug when computing last
+	row constants, in tables that do not start right at the right
+	margin.
+	(org-table-eval-formula): Match and replace remove references.
+	(org-table-formula-substitute-names): Make sure that names inside
+	a "remote" call are left alone, the will be replaced later when
+	the remote call is handled.
+	(org-table-convert-refs-to-rc): Do not convert things that might
+	look like a reference, but are really part of an ID or namei n a
+	remote reference.
+	(org-table-get-remote-range): New function.
+
 2009-01-22  Carsten Dominik  <carsten.dominik@gmail.com>
 
 	* org-faces.el (org-clock-overlay): Fix bug in face definition.

lisp/org-table.el

@@ -1224,7 +1224,7 @@ With prefix ABOVE, insert above the current line."
       (error "Not at a table"))
   (when (eobp) (insert "\n") (backward-character 1))
   (if (not (string-match "|[ \t]*$" (org-current-line-string)))
-      (org-table-align))  
+      (org-table-align))
   (let ((line (org-table-clean-line
 	       (buffer-substring (point-at-bol) (point-at-eol))))
 	(col (current-column)))
@@ -1909,6 +1909,7 @@ For all numbers larger than LIMIT, shift them by DELTA."
 	(if (match-end 1) (push l hlines) (push l dlines))
 	(beginning-of-line 2)
 	(setq l (1+ l)))
+      (push 'hline types) ;; add an imaginary extra hline to the end
       (setq org-table-current-line-types (apply 'vector (nreverse types))
 	    last-dline (car dlines)
 	    org-table-dlines (apply 'vector (cons nil (nreverse dlines)))
@@ -1917,7 +1918,7 @@ For all numbers larger than LIMIT, shift them by DELTA."
       (let* ((l last-dline)
 	     (fields (org-split-string
 		      (buffer-substring (point-at-bol) (point-at-eol))
-		      "|"))
+		      "[ \t]*|[ \t]*"))
 	     (nfields (length fields))
 	     al al2)
 	(loop for i from 1 to nfields do
@@ -2152,6 +2153,16 @@ not overwrite the stored one."
 	(if (and lispp literal) (setq lispp 'literal))
 	;; Check for old vertical references
 	(setq form (org-table-rewrite-old-row-references form))
+	;; Insert remote references
+	(while (string-match "\\<remote([ \t]*\\([-_a-zA-Z0-9]+\\)[ \t]*,[ \t]*\\([^\n)]+\\))" form)
+	  (setq form
+		(replace-match
+		 (save-match-data
+		   (org-table-make-reference
+		    (org-table-get-remote-range
+		     (match-string 1 form) (match-string 2 form))
+		    keep-empty numbers lispp))
+		 t t form)))
 	;; Insert complex ranges
 	(while (and (string-match org-table-range-regexp form)
 		    (> (length (match-string 0 form)) 1))
@@ -2528,12 +2539,14 @@ known that the table will be realigned a little later anyway."
       (setq f (replace-match (concat "$" (cdr a)) t t f)))
     ;; Parameters and constants
     (setq start 0)
-    (while (setq start (string-match "\\$\\([a-zA-Z][_a-zA-Z0-9]*\\)" f start))
-      (setq start (1+ start))
-      (if (setq a (save-match-data
-		    (org-table-get-constant (match-string 1 f))))
-	  (setq f (replace-match
-		   (concat (if pp "(") a (if pp ")")) t t f))))
+    (while (setq start (string-match "\\$\\([a-zA-Z][_a-zA-Z0-9]*\\)\\|\\(\\<remote([^)]*)\\)" f start))
+      (if (match-end 2)
+	  (setq start (match-end 2))
+	(setq start (1+ start))
+	(if (setq a (save-match-data
+		      (org-table-get-constant (match-string 1 f))))
+	    (setq f (replace-match
+		     (concat (if pp "(") a (if pp ")")) t t f)))))
     (if org-table-formula-debug
 	(put-text-property 0 (length f) :orig-formula f1 f))
     f))
@@ -2676,7 +2689,7 @@ Parameters get priority."
 Works for single references, but also for entire formulas and even the
 full TBLFM line."
   (let ((start 0))
-    (while (string-match "\\<\\([a-zA-Z]+\\)\\([0-9]+\\>\\|&\\)\\|\\(;[^\r\n:]+\\)" s start)
+    (while (string-match "\\<\\([a-zA-Z]+\\)\\([0-9]+\\>\\|&\\)\\|\\(;[^\r\n:]+\\|\\<remote([^)]*)\\)" s start)
       (cond
        ((match-end 3)
 	;; format match, just advance
@@ -4068,6 +4081,60 @@ provide ORGTBL directives for the generated table."
 	 (params (org-combine-plists params2 params)))
     (orgtbl-to-generic table params)))
 
+(defun org-table-get-remote-range (name-or-id form)
+  "Get a field value or a list of values in a range from table at ID.
+
+NAME-OR-ID may be the name of a table in the current file as set by
+a \"#+TBLNAME:\" directive.  The first table following this line
+will then be used.  Alternatively, it may be an ID referring to
+any entry, also in a different file.  In this case, the first table
+in that netry will be referenced.
+FORM is a field or range descriptor like \"@2$3\" or or \"B3\" or
+\"@I$2..@II$2\".  All the references must be absolute, not relative.
+
+The return value is either a single string for a single field, or a
+list of the fields in the rectangle ."
+  (save-match-data
+    (let ((id-loc nil)
+	  org-table-column-names org-table-column-name-regexp
+	  org-table-local-parameters org-table-named-field-locations
+	  org-table-current-line-types org-table-current-begin-line
+	  org-table-current-begin-pos org-table-dlines
+	  org-table-hlines org-table-last-alignment
+	  org-table-last-column-widths org-table-last-alignment
+	  org-table-last-column-widths tbeg
+	  buffer loc)
+      (save-excursion
+	(save-restriction
+	  (widen)
+	  (save-excursion
+	    (goto-char (point-min))
+	    (if (re-search-forward
+		 (concat "^#\\+TBLNAME:[ \t]*" (regexp-quote name-or-id) "[ \t]*$")
+		 nil t)
+		(setq buffer (current-buffer) loc (match-beginning 0))
+	      (setq id-loc (org-id-find name-or-id 'marker)
+		    buffer (marker-buffer id-loc)
+		    loc (marker-position id-loc))
+	      (move-marker id-loc nil)))
+	  (switch-to-buffer buffer)
+	  (save-excursion
+	    (save-restriction
+	      (widen)
+	      (goto-char loc)
+	      (forward-char 1)
+	      (unless (and (re-search-forward "^\\(\\*+ \\)\\|[ \t]*|" nil t)
+			   (not (match-beginning 1)))
+		(error "Cannot find a table at NAME or ID %s" name-or-id))
+	      (setq tbeg (point-at-bol))
+	      (org-table-get-specials)
+	      (setq form (org-table-formula-substitute-names form))
+	      (if (and (string-match org-table-range-regexp form)
+		       (> (length (match-string 0 form)) 1))
+		  (save-match-data
+		    (org-table-get-range (match-string 0 form) tbeg 1))
+		form))))))))
+
 (provide 'org-table)
 
 ;; arch-tag: 4d21cfdd-0268-440a-84b0-09237a0fe0ef