Org-babel

<div id="subtitle"> <p>executable source code blocks in org-mode</p> </div> <div id="logo"> <p> <img src="images/tower-of-babel.png" alt="images/tower-of-babel.png"/> <div id="attr"> The Tower of Babel by <a href="http://commons.wikimedia.org/wiki/Pieter_Brueghel_the_Elder" title=""> <b>Pieter Brueghel the Elder</b> </a> </div> <p> And the Lord said, Behold, the people is one, and they have all one language; and this they begin to do; and now nothing will be restrained from them, which they have imagined to do. Genesis 11:1-9 </p> </p> </div>

Introduction

Org-babel is an extension to the very excellent Org-mode, providing the ability to execute source code in many different languages within org-mode documents. The results of code execution — text, tables and graphics — can be integrated into the powerful publishing facilities of org-mode. Org-mode is an Emacs major mode for doing almost anything with plain text. If you are not familiar with Org-mode please take a moment to read the Org-mode homepage before continuing.

Org-babel provides the following modifications to the existing support for blocks of source code examples in the org-mode core.

Interactive source code execution
Arguments to source code blocks
Exportation of source code blocks to files (literate programming)

Getting started

Grab the latest code from the git repo at github/org-babel
```
git clone git://github.com/eschulte/org-babel.git
```
Add the following lines to your .emacs, replacing the path as appropriate. A good place to check that things are up and running would the examples in Basic org-babel functionality.
```
  (add-to-list 'load-path "/path/to/org-babel/lisp")
  (require 'org-babel-init)
```
Finally, activate the subset of supported Org-babel languages which you want to be able to execute on your system. As an example, the following activates python, ruby and R. For a full list of languages, with notes on their dependencies see the Reference / Documentation section below.
```
  (require 'org-babel-python)
  (require 'org-babel-ruby)
  (require 'org-babel-R)
  ;;
  ;; Once you've activated languages, load the library of babel to
  ;; make pre-built helper functions available in the languages you will be using.
  (org-babel-load-library-of-babel)
```

Basic org-babel functionality

Source code blocks

Org-babel is all about source code blocks in org mode. These are blocks of code (in whatever language), that can occur anywhere in an org-mode file. For example, the following is a source block containing ruby code:

#+begin_src ruby
"This file was last evaluated on #{Date.today}"
#+end_src

If you are unfamiliar with the notion of a source code block in org-mode, please have a look at the relevant manual section before proceding.

Note that above is what the source block looks like in the org-mode file. We had to take special steps to make it look that way in the HTML output. Normally, when exported to HTML, source blocks are fontified according to their language, and the begin_src…end_src mark-up is omitted, like this:

"This file was last evaluated on #{Date.today}"

From now on, if you are viewing the HTML version, you will see the HTML output only. However, much of this document consists of interactive examples, and therefore in order to get a feeling for the mechanics of Org-babel it might make most sense to grab the plain text version of this file

org-babel-worg.org

and work through it in Emacs. Alternatively the htmlized version of the plain text of this file at

org-babel-worg.html

allows the plain text version to be viewed (non-interactively) in a web browser.

Source code execution

For interpreted languages such as shell, python, R, etc, org-babel allows source blocks to be executed: the code is passed to the interpreter and you have control over what is done with the results of execution. Here are three examples of code blocks in three different languages, followed by their output. If you are viewing the plain text version of this document in emacs, place point anywhere inside the blocks and use C-c C-c to run the code¹ (and feel free to alter it!).

Ruby

"This file was last evaluated on #{Date.today}"

This file was last evaluated on 2009-08-09

R

matrix(rnorm(6), nrow=2)

-0.138279734486552	-2.2476234005706	-0.0839549402407832
0.0730510956002737	0.0634015508602321	0.174013159381603

ditaa

+---------+
| cBLU    |
|         |
|    +----+
|    |cPNK|
|    |    |
+----+----+

Source code block syntax

The basic syntax of source-code blocks in Org-babel is as follows:

#+srcname: name(arguments)
#+begin_src language header-arguments
body
#+end_src

name: This name is associated with the source-code block. This is similar to the #+tblname lines which can be used to name tables in org-mode files. By referencing the srcname of a source-code block it is possible to evaluate the block from other places, files, or from inside tables.
arguments: Code blocks can have arguments (see below) which are provided using a familiar function-call syntax similar to (e.g.) python or R.
language: The language of the code in the source-code block. Valid values must be members of `org-babel-interpreters'.
header-arguments: Header arguments control many facets of the evaluation and output of source-code blocks. See the Header Arguments section for a complete review of available header arguments.
body: The actual source code which will be evaluated. An important key-binding to become familiar with is C-c '. This calls `org-edit-special' which brings up an edit buffer containing the code using the emacs major mode appropriate to the language.

What happens to the results?

Org-babel provides two fundamentally different modes for capturing the results of code evaluation, specified by the :results header argument.

`:results value` (functional mode)

This means that the 'result' of code evaluation is defined to be the value of the last statement in the block. Thus with this setting, one can view the code block as a function with a return value. And not only can you view it that way, but you can actually use the return value of one source block as input for another (see meta-programming-language). This setting is the default.

As an example, consider the following block of python code and its output.

import time
print("Hello, today's date is %s" % time.ctime())
print('Two plus two is')
2 + 2

Notice that in functional mode, the output consists of the value of the last statement, and nothing else.

`:results output` (scripting mode)

With this setting, org-babel captures all the text output of the code block and places it in the org buffer. One can think of this as a 'scripting' mode: the code block contains a series of commands, and you get the output of all the commands. Unlike in the 'functional' mode, the code block has no return value. (This mode will be more familiar to Sweave users).

Now consider the result of evaluating the same source block as before, but under scripting mode.

import time
print("Hello, today's date is %s" % time.ctime())
print('Two plus two is')
2 + 2

Hello, today's date is Fri Sep  4 19:49:06 2009
Two plus two is

Again, we got what we asked for: all the text output (stdout) from python. Since we didn't print the last value (2 + 2), we didn't get it in our output.

Arguments to source code blocks

In addition to evaluation of code blocks, org-babel allows them to be parameterised (i.e. have arguments). Thus source code blocks now have the status of functions. Arguments to code blocks can be used in both functional and scripting mode.

Simple example of using a source block as a function

First let's look at a very simple example. The following source block defines an org-babel function that will square its input.

x*x

In the org-mode file that looks like this:

#+srcname: square(x)
#+begin_src python
x*x
#+end_src

Now we use the source block:

#+lob: square(x=6)

(for information on the lob syntax see library-of-babel)

A more complex example: using an org-table as input

In this example we're going to define a function to compute a Fibonacci sequence, and we're going to make it take its input from a table in the org-mode buffer.

Here are the inputs for fibonacci-seq:

1	2	3	4	5	6	7	8	9	10
2	4	6	8	10	12	14	16	18	20

in the Org-mode buffer this looks like

#+tblname: fibonacci-inputs
| 1 | 2 | 3 | 4 |  5 |  6 |  7 |  8 |  9 | 10 |
| 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 | 20 |

Emacs Lisp source code

  (defun fibonacci (n)
    (if (or (= n 0) (= n 1))
        n
      (+ (fibonacci (- n 1)) (fibonacci (- n 2)))))
  
  (mapcar (lambda (row)
            (mapcar #'fibonacci row)) fib-inputs)

in the Org-mode buffer this looks like

#+srcname: fibonacci-seq(fib-inputs=fibonacci-inputs)
#+begin_src emacs-lisp
  (defun fibonacci (n)
    (if (or (= n 0) (= n 1))
        n
      (+ (fibonacci (- n 1)) (fibonacci (- n 2)))))
  
  (mapcar (lambda (row)
            (mapcar #'fibonacci row)) fib-inputs)
#+end_src

Results of Emacs Lisp code evaluation

1	1	2	3	5	8	13	21	34	55
1	3	8	21	55	144	377	987	2584	6765

A meta-programming language for org-mode

Since information can pass freely between source-code blocks and org-mode tables you can mix and match languages using each language for those tasks to which it is suited. This makes Org-mode files with Org-babel into a kind of meta-functional programming language in which functions from many languages can work together.

As an example, lets take some system diagnostics in the shell, and then graph them with R.

First we create a code block containing shell code creating a list of the directories in our home directory, together with their sizes. Org-babel automatically converts the output into an org table.

cd ~ && du -sc * |grep -v total

72	"Desktop"
12156104	"Documents"
3482440	"Downloads"
2901720	"Library"
57344	"Movies"
16548024	"Music"
120	"News"
7649472	"Pictures"
0	"Public"
152224	"Sites"
8	"System"
56	"bin"
3821872	"mail"
10605392	"src"
1264	"tools"

Now we use a single line of R code to plot the data as a pie-chart. Note the way that this source block uses the srcname of the previous source block to obtain the data.

pie(dirs[,1], labels = dirs[,2])

Multilingual spreadsheet plugins for org-mode

Not only can Org-babel pass entire tables of data to source code blocks (see arguments-to-source-code-blocks), Org-babel can also be used to call source code blocks from within tables using the Org-mode's existing spreadsheet functionality.

Example 1: data summaries using R

As a simple example, we'll fill in a cell in an org-table with the average value of a few numbers. First, let's make some data. The following source block creates an org table filled with five random numbers between 0 and 1.

runif(n=5, min=0, max=1)

0.850250755203888

0.745323235634714

0.845673063071445

0.761818468105048

0.525476417969912

Now we define a source block to do the calculation we want.

mean(x)

Finally we create the table which is going to make use of the R code. This is done using the sbe ('source block evaluate') macro in the table formula line.

mean
0.533130449522286

To recalculate the table formula, use C-u C-c C-c in the table. Notice that as things stand the calculated value doesn't change, because the data (held in the table above named "tbl-example-data") are static. However, if you delete that data table then the reference will be interpreted as a reference to the source block responsible for generating the data; each time the table formula is recalculated the source block will be evaluated again, and therefore the calculated average value will change.

Example 2: Org-babel test suite

While developing Org-babel, we have used a suite of tests implemented as a large Org-mode table. To run the entire test suite you simply evaluate the table with C-u C-c C-c, and all of the tests are run, the results are compared with expectations, and the table is updated with results and pass/fail statistics.

Here's a sample of our test suite.

functionality	block	arg	expected	results	pass
basic evaluation					pass
emacs lisp	basic-elisp	2	4	4	pass
shell	basic-shell		6	6	pass
ruby	basic-ruby		org-babel	org-babel	pass
python	basic-python		hello world	hello world	pass
R	basic-R		13	13	pass

code blocks for tests

(* 2 n)

expr 1 + 5

date

"org-babel"

'hello world'

b <- 9
b + 4

The Library of Babel

<div id="logo"> <p> <img src="images/library-of-babel-clayette.png" /> <div id="attr"> The Library of Babel, by Pierre Clayette <p> <a href="http://downlode.org/Etext/library_of_babel.html">Full text of the Borges short story</a> </p> </div> </p> </div>

As we saw above with the square example, once a source block function has been defined it can be called using the following short lob notation:

#+lob: square(x=6)

But what about those source code blocks which are so useful you want to have them available in every org-mode buffer?

In addition to the current buffer, Org-babel searches for pre-defined source block functions in the Library of Babel. This is a user-extensible collection of ready-made source-code blocks for handling common tasks. One use for the LoB (not yet done!) will be to provide a choice of data graphing procedures for data held in org-mode tables, using languages such as R, gnuplot, asymptote, etc. If you implement something that might be of use to other org users, please consider adding it to the LoB; similarly, feel free to request help solving a problem using external code via org-babel – there's always a chance that other org users will be able to contribute some helpful code. Org-mode demonstrates that an enormous amount can be achieved using plain text and emacs lisp; the LoB is intended to fill in the gaps.

Org-babel comes pre-populated with the source-code blocks located in the library-of-babel.org file. It is possible to add source-code blocks from any org-mode file to the library by calling

(org-babel-lob-ingest "path/to/file.org")

Note that it is also possible to pass table values or the output of a source-code block to lob functions, and it is possible to reference lob functions in source block arguments.

Reproducible Research

An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.

– D. Donoho

Reproducible Research (RR) is the practice of distributing along with an article of research all data, code, and tools required to reproduce the results discussed in the paper. As such the paper becomes not only a document describing the research but a complete laboratory in which the research can be reproduced and extended.

Org-mode already has exceptional support for exporting to html and LaTeX. Org-babel makes Org-mode a tool for RR by activating the data and source code embedded into Org-mode documents making the entire document executable. This makes it not only possible, but natural to distribute research in a format that encourages readers to recreate your results, and perform their own analysis.

One notable existing RR tool is Sweave which provides for the embedding of R code into LaTeX documents. While Sweave is a mature and very useful tool, we believe that Org-babel has several advantages:

It supports multiple languages (we're not aware of other RR tools that do this)
The export process is flexible and powerful, including HTML as a target in addition to LaTeX
The document can make native use of all the features of Org-mode, such as those for project planning and task management

Literate programming

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

The practitioner of literate programming can be regarded as an essayist, whose main concern is with exposition and excellence of style. Such an author, with thesaurus in hand, chooses the names of variables carefully and explains what each variable means. He or she strives for a program that is comprehensible because its concepts have been introduced in an order that is best for human understanding, using a mixture of formal and informal methods that reinforce each other.

– Donald Knuth

Org-babel supports Literate Programming (LP) by allowing the act of programming to take place inside of Org-mode documents. The Org-mode file can then be exported (woven in LP speak) to html or LaTeX for consumption by a human, and the embedded source code can be extracted (tangled in LP speak) into structured source code files for consumption by a computer.

To support these operations Org-babel relies on Org-mode's existing exporting functionality for weaving of documentation, and on the org-babel-tangle function which makes use of Noweb reference syntax for tangling of code files.

The following example demonstrates the process of tangling in Org-babel.

Simple Literate Programming Example (Noweb syntax)

Tangling functionality is controlled by the tangle family of header-arguments. These arguments can be used to turn tangling on or off (the default) on the source code block, or the outline heading level.

The following demonstrates the combination of three source code blocks into a single source code file using org-babel-tangle.

The following two blocks will not be tangled by default since they have no tangle header arguments.

  echo "/-----------------------------------------------------------\\"

#+srcname: hello-world-prefix
#+begin_src sh :exports none
  echo "/-----------------------------------------------------------\\"
#+end_src

  echo "\-----------------------------------------------------------/"

#+srcname: hello-world-postfix
#+begin_src sh :exports none
  echo "\-----------------------------------------------------------/"
#+end_src

The third block does have a tangle header argument indicating the name of the file to which it should be written. It also has Noweb style references to the two previous source code blocks which will be expanded during tangling to include them in the output file as well.

  # <<hello-world-prefix>>
  echo "|                       hello world                         |"
  # <<hello-world-postfix>>

#+srcname: hello-world
#+begin_src sh :tangle hello :exports none
  # <<hello-world-prefix>>
  echo "|                       hello world                         |"
  # <<hello-world-postfix>>
#+end_src

Calling org-babel-tangle will result in the following being written to the hello.sh file.

  #!/usr/bin/env sh
  # generated by org-babel-tangle
  
  # [[file:~/src/org-babel/org-babel-worg.org::#literate-programming-example][block-16]]
  # <<hello-world-prefix>>
  echo "/-----------------------------------------------------------\\"
  
  echo "|                       hello world                         |"
  # <<hello-world-postfix>>
  echo "\-----------------------------------------------------------/"
  # block-16 ends here

Emacs Initialization with Org-babel

Org-babel has special support for embedding your emacs initialization into Org-mode files. The org-babel-load-file function can be used to load the emacs lisp embedded in a literate Org-mode file in the same way that you might load a regular elisp file.

This allows you to have all the niceness of Org-mode (folding, tags, notes, html export, etc…) available in your emacs initialization.

To try this out either see the simple Literate Emacs Initialization example directly below, or check out the Org-babel Literate Programming version of Phil Hagelberg's excellent emacs-starter-kit available at Org-babel-emacs-starter-kit.

Literate Emacs Initialization

For a simple example of usage follow these 4 steps.

create a directory named .emacs.d in the base of your home directory.
```
mkdir ~/.emacs.d
```

checkout the latest versions of Org-mode and Org-babel into the src subdirectory of this new directory

cd ~/.emacs.d
mkdir src
cd src
git clone git://repo.or.cz/org-mode.git
git clone git://github.com/eschulte/org-babel.git

place the following in a file called init.el in your emacs initialization directory (~/.emacs.d).

  ;;; init.el --- Where all the magic begins
  ;;
  ;; This file loads both
  ;; - Org-mode : http://orgmode.org/ and
  ;; - Org-babel: http://eschulte.github.com/org-babel/
  ;;
  ;; It then loads the rest of our Emacs initialization from Emacs lisp
  ;; embedded in literate Org-mode files.
  
  ;; Load up Org Mode and Org Babel for elisp embedded in Org Mode files
  (setq dotfiles-dir (file-name-directory (or (buffer-file-name) load-file-name)))
  (add-to-list 'load-path (expand-file-name
                           "lisp" (expand-file-name
                                   "org" (expand-file-name
                                          "src" dotfiles-dir))))
  (add-to-list 'load-path (expand-file-name
                           "lisp" (expand-file-name
                                   "org-babel" (expand-file-name
                                                "src" dotfiles-dir))))
  (require 'org-babel-init)
  
  ;; load up all literate org-mode files in this directory
  (mapc #'org-babel-load-file (directory-files dotfiles-dir t "\\.org$"))
  
  ;;; init.el ends here

Implement all of your emacs customizations inside of elisp source-code blocks located in Org-mode files in this directory. They will be loaded by emacs on startup.

Reference / Documentation

Languages

The following can be added to your .emacs and used to activate languages. It includes a brief list of the requirements for each language. Note: this also serves as the list of languages currently supported by Org-babel.

  ;; Uncomment each of the following require lines if you want org-babel
  ;; to support that language.  Each language has a comment explaining
  ;; it's dependencies.  See the related files in lisp/langs for more
  ;; detailed explanations of requirements.
  ;; (require 'org-babel-R)         ;; R and ess-mode
  ;; (require 'org-babel-asymptote) ;; asymptote
  ;; (require 'org-babel-css)       ;; none
  ;; (require 'org-babel-ditaa)     ;; ditaa
  ;; (require 'org-babel-dot)       ;; dot
  ;; (require 'org-babel-gnuplot)   ;; gnuplot, and gnuplot-mode
  ;; (require 'org-babel-haskell)   ;; haskell, haskell-mode, inf-haskell
  ;; (require 'org-babel-ocaml)     ;; ocaml, and tuareg-mode
  ;; (require 'org-babel-python)    ;; python, and python-mode
  ;; (require 'org-babel-ruby)      ;; ruby, irb, ruby-mode, and inf-ruby
  ;; (require 'org-babel-sass)      ;; sass, sass-mode
  ;; (require 'org-babel-sql)       ;; none

Header Arguments

results

results arguments specify what should be done with the output of source-code blocks

The following options are mutually exclusive, and specify how the results should be collected from the source-code block

value

output
The following options are mutually exclusive and specify what type of results the code block will return

vector

specifies that the results should be interpreted as a multidimensional vector (even if the vector is trivial), and will be inserted into the org-mode file as a table

scalar

specifies that the results should be interpreted as a scalar value, and will be inserted into the org-mode file as quoted text

file

specifies that the results should be interpreted as the path to a file, and will be inserted into the org-mode file as a link
The following options specify how the results should be inserted into the org-mode file

replace

the current results replace any previously inserted results from the code block

silent

rather than being inserted into the org-mode file the results are echoed into the message bar

exports

exports arguments specify what should be included in html or latex exports of the org-mode file

code: the body of code is included into the exported file
results: the results of evaluating the code is included in the exported file
both: both the code and results are included in the exported file
none: nothing is included in the exported file

tangle

tangle arguments specify whether or not the source-code block should be included in tangled extraction of source-code files

yes: the source-code block is exported to a source-code file named after the basename (name w/o extension) of the org-mode file
no: (default) the source-code block is not exported to a source-code file
other: any other string passed to the tangle header argument is interpreted as a file basename to which the block will be exported

Noweb reference syntax

The Noweb Literate Programming system allows named blocks of code to be referenced by using a <<code-block-name>> syntax. When a document is tangled these references are replaced with the named code. An example is provided in the literate-programming-example in this document.

Footnotes

Calling C-c C-o on a source-code block will open the block's results in a separate buffer.

org-babel-worg.org 32 KB History Raw