org-mode-mirror/testing

# -*- mode:org -*-
#+TITLE: Org-mode Testing
#+PROPERTY: results silent

* Dependencies

The only dependency is [[http://www.emacswiki.org/emacs/ErtTestLibrary][ERT]] the Emacs testing library which ships with
Emacs24. If you are running an older version of Emacs and don't
already have ERT installed it can be installed from its old [[https://github.com/ohler/ert][git
repository]].

* Non-interactive batch testing from the command line

The simplest way to run the Org-mode test suite is from the command
line with the following invocation. Note that the paths below are
relative to the base of the Org-mode directory.

Also note that many of the current tests uses babel evaluation...

#+BEGIN_SRC sh :dir (expand-file-name "..")
# For Emacs earlier than 24, add -L /path/to/ert
emacs -Q --batch \
-L lisp/ -L testing/ -L testing/lisp -l lisp/org.el \
-l lisp/org-id.el -l testing/org-test.el \
--eval "(progn (org-reload) (setq org-confirm-babel-evaluate nil) \
(org-babel-do-load-languages 'org-babel-load-languages \
'((emacs-lisp . t) (shell . t) (org . t))))" \
-f org-test-run-batch-tests
#+END_SRC

The options in the above command are explained below.

| -Q | ignores any personal configuration ensuring a vanilla Emacs instance is used |
| --batch | runs Emacs in "batch" mode with no gui and termination after execution |
| -l | loads Org-mode and the org mode test suite defined in testing/org-test.el |
| --eval | reloads Org-mode and allows evaluation of code blocks by the tests |
| -f | actually runs the tests using the `org-test-run-batch-tests' function |

* Trigger the tests with 'make'

** Recompile all

Target ~test~ can be used to trigger a test run. The tests start
after cleaning up and recompilation.

#+BEGIN_SRC sh :dir (expand-file-name "..") :results silent
make test
#+END_SRC

See ../mk/default.mk for details.

** Test dirty

The 'dirty' targets are for recompiling without cleaning and
rebuilding everything. This usually speeds up the recompilation
considerably.

The 'dirty' target is called test-dirty.

#+BEGIN_SRC sh :dir (expand-file-name "..") :results silent
make test-dirty
#+END_SRC

Note that the outcome may /not/ be in perfect shape.

* Interactive testing from within Emacs

To run the Org-mode test suite from a current Emacs instance simply
load and run the test suite with the following commands.

1) First load the test suite.
#+BEGIN_SRC emacs-lisp :var here=(buffer-file-name)
(add-to-list 'load-path (file-name-directory here))
(require 'org-test)
#+END_SRC

2) Load required Babel languages
#+BEGIN_SRC emacs-lisp
(org-babel-do-load-languages
'org-babel-load-languages
(and
(mapc (lambda (lang) (add-to-list 'org-babel-load-languages (cons lang t)))
'(emacs-lisp shell org))
org-babel-load-languages))
#+END_SRC

3) Then run the test suite. Babel evaluation confirmation is disabled
and ~C-c C-c~ is enabled while running the tests.
#+BEGIN_SRC emacs-lisp
(let (org-babel-no-eval-on-ctrl-c-ctrl-c
org-confirm-babel-evaluate)
(org-test-run-all-tests))
#+END_SRC

When a test fails, run it interactively and investigate the problem
in the ERT results buffer.

To run one test: Use this as a demo example of a failing test
#+BEGIN_SRC emacs-lisp
(ert-deftest test-org/org-link-escape-ascii-character-demo-of-fail ()
(should (string= "%5B" ; Expecting %5B is correct.
(org-link-escape "[")))
(should (string= "%5C" ; Expecting %5C is wrong, %5D correct.
(org-link-escape "]"))))
#+END_SRC
or evaluate the ~ert-deftest form~ of the test you want to run.
Then ~M-x ert RET
test-org/org-link-escape-ascii-character-demo-of-fail RET~. When
not visible yet switch to the ERT results buffer named ~*ert*~.
When a test failed the ERT results buffer shows the details of the
first ~should~ that failed. See ~(info "(ert)Running Tests
Interactively")~ on how to re-run, start the debugger etc.

To run several tests: ~M-x ert RET "" RET~.

To run all tests of a single test file: ~M-x ert-delete-all-tests
RET~ and confirm. ~M-x load-file RET testing/lisp/.el RET
M-x ert RET t RET~.

Consider to set
#+BEGIN_SRC emacs-lisp
(setq pp-escape-newlines nil)
#+END_SRC
before running the test when looking at ~should~ in the ERT results
buffer. Especially when using ~l~ to look at passed test results
and possibly missing an appropriate setting of ~pp-escape-newlines~
made only temporarily for the running time of the test as
e. g. tests using ~org-test-table-target-expect-tblfm~ do.

* Troubleshooting

- If the variable ~org-babel-no-eval-on-ctrl-c-ctrl-c~ is non-nil then
it will result in some test failure, as there are tests which rely
on this behavior.