Rewrite of the clock table code

* lisp/org.el (org-shorten-string): New function.
* lisp/org-exp.el (org-export-convert-protected-spaces): New function.
(org-export-preprocess-string): Call
`org-export-convert-protected-spaces' to handle new hard spaces.

* lisp/org-clock.el (org-clocktable): New customization group.
(org-clocktable-defaults): New option.
(org-clock-clocktable-formatter): New option.
(org-clock-clocktable-default-properties): New option.
(org-dblock-write:clocktable): Rewrite to split out functionality
into separate functions.
(org-clocktable-write-default):
(org-clocktable-indent-string):
(org-clock-get-table-data): New functions.
* lisp/org-agenda.el (org-agenda-list):
(org-agenda-redo):
(org-agenda-clockreport-mode):
(org-agenda-set-mode-name): Rewrite to implement filtered clock tables.
* doc/org.texi (Clocking commands):
(The clock table): New sections.
(Agenda commands): Document filtered clock reports.

doc/org.texi

@@ -260,7 +260,6 @@ Dates and times
 * Creating timestamps::         Commands which insert timestamps
 * Deadlines and scheduling::    Planning your work
 * Clocking work time::          Tracking how long you spend on a task
-* Resolving idle time::         Resolving time if you've been idle
 * Effort estimates::            Planning work effort in advance
 * Relative timer::              Notes with a running timer
 * Countdown timer::             Starting a countdown timer for a task
@@ -275,6 +274,12 @@ Deadlines and scheduling
 * Inserting deadline/schedule::  Planning items
 * Repeated tasks::              Items that show up again and again
 
+Clocking work time
+
+* Clocking commands::           Starting and stopping a clock
+* The clock table::             Detailed reports
+* Resolving idle time::         Resolving time when you've been idle
+
 Capture - Refile - Archive
 
 * Capture::                     Capturing new stuff
@@ -5006,7 +5011,6 @@ is used in a much wider sense.
 * Creating timestamps::         Commands which insert timestamps
 * Deadlines and scheduling::    Planning your work
 * Clocking work time::          Tracking how long you spend on a task
-* Resolving idle time::         Resolving time if you've been idle
 * Effort estimates::            Planning work effort in advance
 * Relative timer::              Notes with a running timer
 * Countdown timer::             Starting a countdown timer for a task
@@ -5547,8 +5551,10 @@ subtree, with dates shifted in each copy.  The command @kbd{C-c C-x c} was
 created for this purpose, it is described in @ref{Structure editing}.
 
 
-@node Clocking work time, Resolving idle time, Deadlines and scheduling, Dates and Times
+@node Clocking work time, Effort estimates, Deadlines and scheduling, Dates and Times
 @section Clocking work time
+@cindex clocking time
+@cindex time clocking
 
 Org-mode allows you to clock the time you spend on specific tasks in a
 project.  When you start working on an item, you can start the clock.
@@ -5569,6 +5575,15 @@ on this task while outside Emacs, use @code{(setq org-clock-persist t)}.}
 will be found (@pxref{Resolving idle time}) and you will be prompted about
 what to do with it.
 
+@menu
+* Clocking commands::           Starting and stopping a clock
+* The clock table::             Detailed reports
+* Resolving idle time::         Resolving time when you've been idle
+@end menu
+
+@node Clocking commands, The clock table, Clocking work time, Clocking work time
+@subsection Clocking commands
+
 @table @kbd
 @kindex C-c C-x C-i
 @item C-c C-x C-i
@@ -5641,6 +5656,22 @@ recorded under that heading, including the time of any subheadings. You
 can use visibility cycling to study the tree, but the overlays disappear
 when you change the buffer (see variable
 @code{org-remove-highlights-with-change}) or press @kbd{C-c C-c}.
+@end table
+
+The @kbd{l} key may be used in the timeline (@pxref{Timeline}) and in
+the agenda (@pxref{Weekly/daily agenda}) to show which tasks have been
+worked on or closed during a day.
+
+@node The clock table, Resolving idle time, Clocking commands, Clocking work time
+@subsection The clock table
+@cindex clocktable, dynamic block
+@cindex report, of clocked time
+
+Org mode can produce quite complex reports based on the time clocking
+inormation.  Such a report is called a @emph{clock table}, because it is
+formatted as one or several Org tables.
+
+@table @kbd
 @kindex C-c C-x C-r
 @item C-c C-x C-r
 Insert a dynamic block (@pxref{Dynamic blocks}) containing a clock
@@ -5648,17 +5679,45 @@ report as an Org-mode table into the current file.  When the cursor is
 at an existing clock table, just update it.  When called with a prefix
 argument, jump to the first clock report in the current document and
 update it.
+@kindex C-c C-c
+@item C-c C-c
+@kindex C-c C-x C-u
+@itemx C-c C-x C-u
+Update dynamic block at point.  The cursor needs to be in the
+@code{#+BEGIN} line of the dynamic block.
+@kindex C-u C-c C-x C-u
+@item C-u C-c C-x C-u
+Update all dynamic blocks (@pxref{Dynamic blocks}).  This is useful if
+you have several clock table blocks in a buffer.
+@kindex S-@key{left}
+@kindex S-@key{right}
+@item S-@key{left}
+@itemx S-@key{right}
+Shift the current @code{:block} interval and update the table.  The cursor
+needs to be in the @code{#+BEGIN: clocktable} line for this command.  If
+@code{:block} is @code{today}, it will be shifted to @code{today-1} etc.
+@end table
+
+
+Here is an example of the frame for a clock table as it is inserted into the
+buffer with the @kbd{C-c C-x C-r} command:
+
 @cindex #+BEGIN, clocktable
 @example
 #+BEGIN: clocktable :maxlevel 2 :emphasize nil :scope file
 #+END: clocktable
 @end example
 @noindent
-If such a block already exists at point, its content is replaced by the
-new table.  The @samp{BEGIN} line can specify options:
+@vindex org-clocktable-defaults
+The @samp{BEGIN} line and specify a number of options to define the scope,
+structure, and formatting of the report.  Defaults for all these options can
+be configured in the variable @code{org-clocktable-defaults}.
+
+@noindent First there are options that determine which clock entries are to
+be selected:
 @example
 :maxlevel    @r{Maximum level depth to which times are listed in the table.}
-:emphasize   @r{When @code{t}, emphasize level one and level two items.}
+             @r{Clocks at deeper levels will be summed into the upper level.}
 :scope       @r{The scope to consider.  This can be any of the following:}
              nil        @r{the current buffer or narrowed region}
              file       @r{the full current buffer}
@@ -5685,15 +5744,34 @@ new table.  The @samp{BEGIN} line can specify options:
 :tend        @r{A time string specifying when to stop considering times.}
 :step        @r{@code{week} or @code{day}, to split the table into chunks.}
              @r{To use this, @code{:block} or @code{:tstart}, @code{:tend} are needed.}
-:stepskip0   @r{Don't show steps that have zero time}
-:tags        @r{A tags match to select entries that should contribute}
+:stepskip0   @r{Do not show steps that have zero time.}
+:fileskip0   @r{Do not show table sections from files which did not contribute.}
+:tags        @r{A tags match to select entries that should contribute}.
+@end example
+
+Then there are options which determine the formatting of the table.  There
+options are interpreted by the function @code{org-clocktable-write-default},
+but you can specify your own function using the @code{:formatter} parameter.
+@example
+:emphasize   @r{When @code{t}, emphasize level one and level two items.}
 :link        @r{Link the item headlines in the table to their origins.}
+:narrow      @r{An integer to limit the width of the headline column in}
+             @r{the org table.  Does not work together with @code{:link}.}
+             @r{If you write it like @samp{50!}, then the headline will also}
+             @r{be shortened in export, and will work with @code{:link}.}
+:indent      @r{Indent each headline field according to its level.}
+:tcolumns    @r{Number of columns to be used for times.  If this is smaller}
+             @r{than @code{:maxlevel}, lower levels will be lumped into one column.}
+:level       @r{Should a level number column be included?}
+:compact     @r{Abbreviation for @code{:level nil :indent t :narrow 40! :tcolumns 1}}
+             @r{All are overwritten except if there is an explicit @code{:narrow}}
+:timestamp   @r{A timestamp for the entry, when available.  Look for SCHEDULED,}
+             @r{DEADLINE, TIMESTAMP and TIMESTAMP_IA, in this order.}
 :formula     @r{Content of a @code{#+TBLFM} line to be added and evaluated.}
              @r{As a special case, @samp{:formula %} adds a column with % time.}
              @r{If you do not specify a formula here, any existing formula.}
              @r{below the clock table will survive updates and be evaluated.}
-:timestamp   @r{A timestamp for the entry, when available.  Look for SCHEDULED,}
-             @r{DEADLINE, TIMESTAMP and TIMESTAMP_IA, in this order.}
+:formatter   @r{A function to format clock data and insert it into the buffer.}
 @end example
 To get a clock summary of the current level 1 tree, for the current
 day, you could write
@@ -5715,31 +5793,15 @@ A summary of the current subtree with % times would be
 #+BEGIN: clocktable :scope subtree :link t :formula %
 #+END: clocktable
 @end example
-@kindex C-c C-c
-@item C-c C-c
-@kindex C-c C-x C-u
-@itemx C-c C-x C-u
-Update dynamic block at point.  The cursor needs to be in the
-@code{#+BEGIN} line of the dynamic block.
-@kindex C-u C-c C-x C-u
-@item C-u C-c C-x C-u
-Update all dynamic blocks (@pxref{Dynamic blocks}).  This is useful if
-you have several clock table blocks in a buffer.
-@kindex S-@key{left}
-@kindex S-@key{right}
-@item S-@key{left}
-@itemx S-@key{right}
-Shift the current @code{:block} interval and update the table.  The cursor
-needs to be in the @code{#+BEGIN: clocktable} line for this command.  If
-@code{:block} is @code{today}, it will be shifted to @code{today-1} etc.
-@end table
-
-The @kbd{l} key may be used in the timeline (@pxref{Timeline}) and in
-the agenda (@pxref{Weekly/daily agenda}) to show which tasks have been
-worked on or closed during a day.
+A horizontally compact representation of everything clocked during last week
+would be
+@example
+#+BEGIN: clocktable :block lastweek :indent t :narrow 40! :tcolumn 1
+#+END: clocktable
+@end example
 
-@node Resolving idle time, Effort estimates, Clocking work time, Dates and Times
-@section Resolving idle time
+@node Resolving idle time,  , The clock table, Clocking work time
+@subsection Resolving idle time
 @cindex resolve idle time
 
 @cindex idle, resolve, dangling
@@ -5806,7 +5868,7 @@ to a recovery event rather than a set amount of idle time.
 You can also check all the files visited by your Org agenda for dangling
 clocks at any time using @kbd{M-x org-resolve-clocks}.
 
-@node Effort estimates, Relative timer, Resolving idle time, Dates and Times
+@node Effort estimates, Relative timer, Clocking work time, Dates and Times
 @section Effort estimates
 @cindex effort estimates
 
@@ -7584,7 +7646,10 @@ Toggle Clockreport mode.  In Clockreport mode, the daily/weekly agenda will
 always show a table with the clocked times for the timespan and file scope
 covered by the current agenda view.  The initial setting for this mode in new
 agenda buffers can be set with the variable
-@code{org-agenda-start-with-clockreport-mode}.
+@code{org-agenda-start-with-clockreport-mode}.  By using a prefix argument
+when toggling this mode (i.e. @kbd{C-u R}), the clock table will not show
+contributions from entries that are hidden by agenda filtering@footnote{Only
+tags filtering will be respected here, effort filtering is ignored.}.
 @c
 @kindex v E
 @kindex E

lisp/org-agenda.el

@@ -2778,6 +2778,7 @@ removed from the entry content.  Currently only `planning' is allowed here."
 (defvar org-agenda-columns-active nil)
 (defvar org-agenda-name nil)
 (defvar org-agenda-filter nil)
+(defvar org-agenda-filter-while-redo nil)
 (defvar org-agenda-filter-preset nil
   "A preset of the tags filter used for secondary agenda filtering.
 This must be a list of strings, each string must be a single tag preceded
@@ -3299,7 +3300,7 @@ given in `org-agenda-start-on-weekday'."
 	 (day-cnt 0)
 	 (inhibit-redisplay (not debug-on-error))
 	 s e rtn rtnall file date d start-pos end-pos todayp nd wd
-	 clocktable-start clocktable-end)
+	 clocktable-start clocktable-end filter)
     (setq org-agenda-redo-command
 	  (list 'org-agenda-list (list 'quote include-all) start-day ndays))
     ;; Make the list of days
@@ -3425,6 +3426,15 @@ given in `org-agenda-start-on-weekday'."
 	(setq p (plist-put p :tstart clocktable-start))
 	(setq p (plist-put p :tend clocktable-end))
 	(setq p (plist-put p :scope 'agenda))
+	(when (and (eq org-agenda-clockreport-mode 'with-filter)
+		   (setq filter (or org-agenda-filter-while-redo
+				    (get 'org-agenda-filter :preset-filter))))
+	  (setq p (plist-put p :tags (mapconcat (lambda (x)
+						  (if (string-match "[<>=]" x)
+						      ""
+						    x))
+						filter ""))))
+	(message "%s" (plist-get p :tags)) (sit-for 2)
 	(setq tbl (apply 'org-get-clocktable p))
 	(insert tbl)))
     (goto-char (point-min))
@@ -5567,6 +5577,7 @@ When this is the global TODO list, a prefix argument will be interpreted."
   (let* ((org-agenda-keep-modes t)
 	 (filter org-agenda-filter)
 	 (preset (get 'org-agenda-filter :preset-filter))
+	 (org-agenda-filter-while-redo (or filter preset))
 	 (cols org-agenda-columns-active)
 	 (line (org-current-line))
 	 (window-line (- line (org-current-line (window-start))))
@@ -6094,11 +6105,15 @@ so that the date SD will be in that range."
 	   (if org-agenda-entry-text-mode "on" "off")
 	   (if (integerp arg) arg org-agenda-entry-text-maxlines)))
 
-(defun org-agenda-clockreport-mode ()
-  "Toggle clocktable mode in an agenda buffer."
-  (interactive)
+(defun org-agenda-clockreport-mode (&optional with-filter)
+  "Toggle clocktable mode in an agenda buffer.
+With prefix arg WITH-FILTER, make the clocktable respect the current
+agenda filter."
+  (interactive "P")
   (org-agenda-check-type t 'agenda)
-  (setq org-agenda-clockreport-mode (not org-agenda-clockreport-mode))
+  (if with-filter
+      (setq org-agenda-clockreport-mode 'with-filter)
+    (setq org-agenda-clockreport-mode (not org-agenda-clockreport-mode)))
   (org-agenda-set-mode-name)
   (org-agenda-redo)
   (message "Clocktable mode is %s"
@@ -6199,7 +6214,10 @@ When called with a prefix argument, include all archive files as well."
 			" Archives"
 		      (format " :%s:" org-archive-tag))
 		  "")
-		(if org-agenda-clockreport-mode " Clock"   "")))
+		(if org-agenda-clockreport-mode
+		    (if (eq org-agenda-clockreport-mode 'with-filter)
+			" Clock{}" " Clock")
+		  "")))
   (force-mode-line-update))
 
 (defun org-agenda-post-command-hook ()

lisp/org-clock.el

@@ -34,7 +34,7 @@
 (eval-when-compile
   (require 'cl))
 
-(declare-function calendar-absolute-from-iso    "cal-iso"    (&optional date))
+(declare-function calendar-absolute-from-iso "cal-iso" (&optional date))
 (declare-function notifications-notify "notifications" (&rest params))
 (defvar org-time-stamp-formats)
 
@@ -222,11 +222,48 @@ string as argument."
 	  (string :tag "Program")
 	  (function :tag "Function")))
 
-(defcustom org-clock-clocktable-default-properties '(:maxlevel 2 :scope file)
-  "Default properties for new clocktables."
+(defgroup org-clocktable nil
+  "Options concerning the clock table in Org-mode."
+  :tag "Org Clock Table"
+  :group 'org-clock)
+
+(defcustom org-clocktable-defaults
+  (list
+   :maxlevel 2
+   :scope 'file
+   :block nil
+   :tstart nil
+   :tend nil
+   :step nil
+   :stepskip0 nil
+   :fileskip0 nil
+   :tags nil
+   :emphasize nil
+   :link nil
+   :narrow '40!
+   :indent t
+   :formula nil
+   :timestamp nil
+   :level nil
+   :tcolumns nil
+   :formatter nil)
+  "Default properties for clock tables."
   :group 'org-clock
   :type 'plist)
 
+(defcustom org-clock-clocktable-formatter 'org-clocktable-write-default
+  "Function to turn clockng data into a table.
+For more information, see `org-clocktable-write-default'."
+  :group 'org-clocktable
+  :type 'function)
+
+(defcustom org-clock-clocktable-default-properties '(:maxlevel 2 :scope file)
+  "Default properties for new clocktables.
+These will be inserted into the BEGIN line, to make it easy for users to
+play with them."
+  :group 'org-clocktable
+  :type 'plist)
+
 (defcustom org-clock-idle-time nil
   "When non-nil, resolve open clocks if the user is idle more than X minutes."
   :group 'org-clock
@@ -1755,48 +1792,35 @@ the currently selected interval size."
 
 (defun org-dblock-write:clocktable (params)
   "Write the standard clocktable."
+  (setq params (org-combine-plists org-clocktable-defaults params))
   (catch 'exit
-    (let* ((hlchars '((1 . "*") (2 . "/")))
-	   (ins (make-marker))
-	   (total-time nil)
-	   (scope (plist-get params :scope))
-	   (tostring (plist-get  params :tostring))
-	   (multifile (plist-get  params :multifile))
-	   (header (plist-get  params :header))
-	   (maxlevel (or (plist-get params :maxlevel) 3))
-	   (step (plist-get params :step))
-	   (emph (plist-get params :emphasize))
-	   (timestamp (plist-get params :timestamp))
+    (let* ((scope (plist-get params :scope))
+	   (block (plist-get params :block))
 	   (ts (plist-get params :tstart))
 	   (te (plist-get params :tend))
-	   (block (plist-get params :block))
 	   (link (plist-get params :link))
-	   (tags (plist-get params :tags))
-	   (matcher (if tags (cdr (org-make-tags-matcher tags))))
-	   ipos time p level hlc hdl tsp props content recalc formula pcol
-	   cc beg end pos tbl tbl1 range-text rm-file-column scope-is-list st)
-      (setq org-clock-file-total-minutes nil)
+	   (maxlevel (or (plist-get params :maxlevel) 3))
+	   (step (plist-get params :step))
+	   (timestamp (plist-get params :timestamp))
+	   (formatter (or (plist-get params :formatter)
+			  org-clock-clocktable-formatter
+			  'org-clocktable-write-default))
+	   cc range-text ipos pos one-file-with-archives
+	   scope-is-list tbls level link)
+
+      ;; Check if we need to do steps
+      (when block
+	;; Get the range text for the header
+	(setq cc (org-clock-special-range block nil t)
+	      ts (car cc) te (nth 1 cc) range-text (nth 2 cc)))
       (when step
+	;; Write many tables, in steps
 	(unless (or block (and ts te))
 	  (error "Clocktable `:step' can only be used with `:block' or `:tstart,:end'"))
 	(org-clocktable-steps params)
 	(throw 'exit nil))
-      (when block
-	(setq cc (org-clock-special-range block nil t)
-	      ts (car cc) te (nth 1 cc) range-text (nth 2 cc)))
-      (when (integerp ts) (setq ts (calendar-gregorian-from-absolute ts)))
-      (when (integerp te) (setq te (calendar-gregorian-from-absolute te)))
-      (when (and ts (listp ts))
-	(setq ts (format "%4d-%02d-%02d" (nth 2 ts) (car ts) (nth 1 ts))))
-      (when (and te (listp te))
-	(setq te (format "%4d-%02d-%02d" (nth 2 te) (car te) (nth 1 te))))
-      ;; Now the times are strings we can parse.
-      (if ts (setq ts (org-float-time
-		       (apply 'encode-time (org-parse-time-string ts)))))
-      (if te (setq te (org-float-time
-		       (apply 'encode-time (org-parse-time-string te)))))
-      (move-marker ins (point))
-      (setq ipos (point))
+
+      (setq ipos (point)) ; remember the insertion position
 
       ;; Get the right scope
       (setq pos (point))
@@ -1810,166 +1834,251 @@ the currently selected interval size."
 	(setq scope (org-add-archive-files scope)))
        ((eq scope 'file-with-archives)
 	(setq scope (org-add-archive-files (list (buffer-file-name)))
-	      rm-file-column t)))
+	      one-file-with-archives t)))
       (setq scope-is-list (and scope (listp scope)))
-      (save-restriction
-	(cond
-	 ((not scope))
-	 ((eq scope 'file) (widen))
-	 ((eq scope 'subtree) (org-narrow-to-subtree))
-	 ((eq scope 'tree)
-	  (while (org-up-heading-safe))
-	  (org-narrow-to-subtree))
-	 ((and (symbolp scope) (string-match "^tree\\([0-9]+\\)$"
-					     (symbol-name scope)))
-	  (setq level (string-to-number (match-string 1 (symbol-name scope))))
-	  (catch 'exit
-	    (while (org-up-heading-safe)
-	      (looking-at outline-regexp)
-	      (if (<= (org-reduced-level (funcall outline-level)) level)
-		  (throw 'exit nil))))
-	  (org-narrow-to-subtree))
-	 (scope-is-list
+      (if scope-is-list
+	  ;; we collect from several files
 	  (let* ((files scope)
-		 (scope 'agenda)
-		 (p1 (copy-sequence params))
 		 file)
-	    (setq p1 (plist-put p1 :tostring t))
-	    (setq p1 (plist-put p1 :multifile t))
-	    (setq p1 (plist-put p1 :scope 'file))
 	    (org-prepare-agenda-buffers files)
 	    (while (setq file (pop files))
 	      (with-current-buffer (find-buffer-visiting file)
-		(setq org-clock-file-total-minutes 0)
-		(setq tbl1 (org-dblock-write:clocktable p1))
-		(when tbl1
-		  (push (org-clocktable-add-file
-			 file
-			 (concat "| |*File time*|*"
-				 (org-minutes-to-hh:mm-string
-				  org-clock-file-total-minutes)
-				 "*|\n"
-				 tbl1)) tbl)
-		  (setq total-time (+ (or total-time 0)
-				      org-clock-file-total-minutes))))))))
-	(goto-char pos)
-
-	(unless scope-is-list
-	  (org-clock-sum ts te
-			 (unless (null matcher)
-			   (lambda ()
-			     (let ((tags-list
-				    (org-split-string
-				     (or (org-entry-get (point) "ALLTAGS") "")
-				     ":")))
-			       (eval matcher)))))
-	  (goto-char (point-min))
-	  (setq st t)
-	  (while (or (and (bobp) (prog1 st (setq st nil))
-			  (get-text-property (point) :org-clock-minutes)
-			  (setq p (point-min)))
-		     (setq p (next-single-property-change (point) :org-clock-minutes)))
-	    (goto-char p)
-	    (when (setq time (get-text-property p :org-clock-minutes))
-	      (save-excursion
-		(beginning-of-line 1)
-		(when (and (looking-at (org-re "\\(\\*+\\)[ \t]+\\(.*?\\)\\([ \t]+:[[:alnum:]_@#%:]+:\\)?[ \t]*$"))
-			   (setq level (org-reduced-level
-					(- (match-end 1) (match-beginning 1))))
-			   (<= level maxlevel))
-		  (setq hlc (if emph (or (cdr (assoc level hlchars)) "") "")
-			hdl (if (not link)
-				(match-string 2)
-			      (org-make-link-string
-			       (format "file:%s::%s"
-				       (buffer-file-name)
-				       (save-match-data
-					 (org-make-org-heading-search-string
-					  (match-string 2))))
-			       (match-string 2)))
-			tsp (when timestamp
-			      (setq props (org-entry-properties (point)))
-			      (or (cdr (assoc "SCHEDULED" props))
-				  (cdr (assoc "TIMESTAMP" props))
-				  (cdr (assoc "DEADLINE" props))
-				  (cdr (assoc "TIMESTAMP_IA" props)))))
-		  (if (and (not multifile) (= level 1)) (push "|-" tbl))
-		  (push (concat
-			 "| " (int-to-string level) "|"
-			 (if timestamp (concat tsp "|") "")
-			 hlc hdl hlc " |"
-			 (make-string (1- level) ?|)
-			 hlc (org-minutes-to-hh:mm-string time) hlc
-			 " |") tbl))))))
-	(setq tbl (nreverse tbl))
-	(if tostring
-	    (if tbl (mapconcat 'identity tbl "\n") nil)
-	  (goto-char ins)
-	  (insert-before-markers
-	   (or header
-	       (concat
-		"Clock summary at ["
-		(substring
-		 (format-time-string (cdr org-time-stamp-formats))
-		 1 -1)
-		"]"
-		(if block (concat ", for " range-text ".") "")
-		"\n\n"))
-	   (if scope-is-list "|File" "")
-	   "|L|" (if timestamp "Timestamp|" "") "Headline|Time|\n")
-	  (setq total-time (or total-time org-clock-file-total-minutes))
-	  (insert-before-markers
-	   "|-\n|"
-	   (if scope-is-list "|" "")
-	   (if timestamp "|Timestamp|" "|")
-	   "*Total time*| *"
-	   (org-minutes-to-hh:mm-string (or total-time 0))
-	   "*|\n|-\n")
-	  (setq tbl (delq nil tbl))
-	  (if (and (stringp (car tbl)) (> (length (car tbl)) 1)
-		   (equal (substring (car tbl) 0 2) "|-"))
-	      (pop tbl))
-	  (insert-before-markers (mapconcat
-				  'identity (delq nil tbl)
-				  (if scope-is-list "\n|-\n" "\n")))
-	  (backward-delete-char 1)
-	  (if (setq formula (plist-get params :formula))
-	      (cond
-	       ((eq formula '%)
-		(setq pcol (+ (if scope-is-list 1 0) maxlevel 3))
-		(insert
-		 (format
-		  "\n#+TBLFM: $%d='(org-clock-time%% @%d$%d $%d..$%d);%%.1f"
-		  pcol
-		  2
-		  (+ 3 (if scope-is-list 1 0))
-		  (+ (if scope-is-list 1 0) 3)
-		  (1- pcol)))
-		(setq recalc t))
-	       ((stringp formula)
-		(insert "\n#+TBLFM: " formula)
-		(setq recalc t))
-	       (t (error "invalid formula in clocktable")))
-	    ;; Should we rescue an old formula?
-	    (when (stringp (setq content (plist-get params :content)))
-	      (when (string-match "^\\([ \t]*#\\+TBLFM:.*\\)" content)
-		(setq recalc t)
-		(insert "\n" (match-string 1 (plist-get params :content)))
-		(beginning-of-line 0))))
-	  (goto-char ipos)
-	  (skip-chars-forward "^|")
-	  (org-table-align)
-	  (when recalc
-	    (if (eq formula '%)
-		(save-excursion (org-table-goto-column pcol nil 'force)
-				(insert "%")))
-	    (org-table-recalculate 'all))
-	  (when rm-file-column
-	    (forward-char 1)
-	    (org-table-delete-column))
-	  total-time)))))
+		(save-excursion
+		  (save-restriction
+		    (push (org-clock-get-table-data file params) tbls))))))
+	;; Just from the current file
+	(save-restriction
+	  ;; get the right range into the restriction
+	  (org-prepare-agenda-buffers (list (buffer-file-name)))
+	  (cond
+	   ((not scope))  ; use the restriction as it is now
+	   ((eq scope 'file) (widen))
+	   ((eq scope 'subtree) (org-narrow-to-subtree))
+	   ((eq scope 'tree)
+	    (while (org-up-heading-safe))
+	    (org-narrow-to-subtree))
+	   ((and (symbolp scope) (string-match "^tree\\([0-9]+\\)$"
+					       (symbol-name scope)))
+	    (setq level (string-to-number (match-string 1 (symbol-name scope))))
+	    (catch 'exit
+	      (while (org-up-heading-safe)
+		(looking-at outline-regexp)
+		(if (<= (org-reduced-level (funcall outline-level)) level)
+		    (throw 'exit nil))))
+	    (org-narrow-to-subtree)))
+	  ;; do the table, with no file name.
+	  (push (org-clock-get-table-data nil params) tbls)))
+
+      ;; OK, at this point we tbls as a list of tables, one per file
+      (setq tbls (nreverse tbls))
+
+      (setq params (plist-put params :multifile scope-is-list))
+      (setq params (plist-put params :one-file-with-archives
+			      one-file-with-archives))
+
+      (funcall formatter ipos tbls params))))
+
+(defun org-clocktable-write-default (ipos tables params)
+  "Write out a clock table at position IPOS in the current buffer
+TABLES is a list of tables with clocking data as produced by
+`org-clock-get-table-data'.  PARAMS is the parameter property list obtained
+from the dynamic block defintion."
+  ;; This function looks quite complicated, mainly because there are a lot
+  ;; of options which can add or remove columns.  I have massively commented
+  ;; function, to I hope it is understandable.  If someone want to write
+  ;; there own special formatter, this maybe much easier because there can
+  ;; be a fixed format with a well-defined number of columns...
+  (let* ((hlchars '((1 . "*") (2 . "/")))
+	 (multifile (plist-get params :multifile))
+	 (block (plist-get params :block))
+	 (ts (plist-get params :tstart))
+	 (te (plist-get params :tend))
+	 (header (plist-get  params :header))
+	 (narrow (plist-get params :narrow))
+	 (maxlevel (or (plist-get params :maxlevel) 3))
+	 (emph (plist-get params :emphasize))
+	 (level-p (plist-get params :level))
+	 (timestamp (plist-get params :timestamp))
+	 (ntcol (max 1 (or (plist-get params :tcolumns) 100)))
+	 (rm-file-column (plist-get params :one-file-with-archives))
+	 (indent (plist-get params :indent))
+	 link range-text total-time tbl level hlc formula pcol
+	 recalc content narrow-cut-p)
+
+    ;; Implement abbreviations
+    (when (plist-get params :compact)
+      (setq level nil indent t narrow (or narrow '40!) ntcol 1))
+
+    ;; Some consistency test for parameters
+      (unless (integerp ntcol)
+	(setq params (plist-put params :tcolumns (setq ntcol 100))))
+
+      (when (and narrow (integerp narrow) link)
+	;; We cannot have both integer narrow and link
+	(message
+	 "Suppressing :narrow INTEGER in clocktable because :link was also given")
+	(setq narrow nil))
+
+      (when narrow
+	(cond
+	 ((integerp narrow))
+	 ((and (symbolp narrow)
+	       (string-match "\\`[0-9]+!\\'" (symbol-name narrow)))
+	  (setq narrow-cut-p t
+		narrow (string-to-number (substring (symbol-name narrow) 0 -1))))
+	 (t
+	  (error "Invalid value %s of :narrow property in clock table" narrow))))
+
+      (when block
+	;; Get the range text for the header
+	(setq range-text (nth 2 (org-clock-special-range block nil t))))
+
+      ;; Compute the total time
+      (setq total-time (apply '+ (mapcar 'cadr tables)))
+
+      ;; Now we need to output this tsuff
+      (goto-char ipos)
+
+      ;; Insert the text *before* the actual table
+      (insert-before-markers
+       (or header
+	   ;; Format the standard header
+	   (concat
+	    "Clock summary at ["
+	    (substring
+	     (format-time-string (cdr org-time-stamp-formats))
+	     1 -1)
+	    "]"
+	    (if block (concat ", for " range-text ".") "")
+	    "\n\n")))
+
+      ;; Insert the narrowing line
+      (when (and narrow (integerp narrow) (not narrow-cut-p))
+	(insert-before-markers
+	 "|"                            ; table line starter
+	 (if multifile "|" "")          ; file column, maybe
+	 (if level-p   "|" "")          ; level column, maybe
+	 (if timestamp "|" "")          ; timestamp column, maybe
+	 (format "<%d>| |\n" narrow)))  ; headline and time columns
+
+      ;; Insert the table header line
+      (insert-before-markers
+       "|"                              ; table line starter
+       (if multifile "File|"      "")   ; file column, maybe
+       (if level-p   "L|"         "")   ; level column, maybe
+       (if timestamp "Timestamp|" "")   ; timestamp column, maybe
+       "Headline|Time|\n")              ; headline and time columns
+
+      ;; Insert the total time in the table
+      (insert-before-markers
+       "|-\n"                           ; a hline
+       "|"                              ; table line starter
+       (if multifile "| ALL " "")       ; file column, maybe
+       (if level-p   "|"      "")       ; level column, maybe
+       (if timestamp "|"      "")       ; timestamp column, maybe
+       "*Total time*| "                 ; instead of a headline
+       "*"
+       (org-minutes-to-hh:mm-string (or total-time 0)) ; the time
+       "*|\n")                          ; close line
+
+      ;; Now iterate over the tables and insert the data
+      ;; but only if any time has been collected
+      (when (and total-time (> total-time 0))
+
+	(while (setq tbl (pop tables))
+	  ;; now tbl is the table resulting from one file.
+	  (setq file-time (nth 1 tbl))
+	  (when (or (and file-time (> file-time 0))
+		    (not (plist-get params :fileskip0)))
+	    (insert-before-markers "|-\n")  ; a hline because a new file starts
+	    ;; First the file time, if we have multiple files
+	    (when multifile
+	      ;; Summarize the time colleted from this file
+	      (insert-before-markers
+	       (format "| %s %s | %s*File time* | *%s*|\n"
+		       (file-name-nondirectory (car tbl))
+		       (if level-p   "| " "") ; level column, maybe
+		       (if timestamp "| " "") ; timestamp column, maybe
+		       (org-minutes-to-hh:mm-string (nth 1 tbl))))) ; the time
+
+	    ;; Get the list of node entries and iterate over it
+	    (setq entries (nth 2 tbl))
+	    (while (setq entry (pop entries))
+	      (setq level (car entry)
+		    headline (nth 1 entry)
+		    hlc (if emph (or (cdr (assoc level hlchars)) "") ""))
+	      (if narrow-cut-p
+		  (setq headline (org-shorten-string headline narrow)))
+	      (insert-before-markers
+	       "|"                      ; start the table line
+	       (if multifile "|" "")    ; free space for file name column?
+	       (if level-p (format "%d|" (car entry)) "")   ; level, maybe
+	       (if timestamp (concat (nth 2 entry) "|") "") ; timestamp, maybe
+	       (if indent (org-clocktable-indent-string level) "") ; indentation
+	       hlc headline hlc "|"                                ; headline
+	       (make-string (min (1- ntcol) (or (- level 1))) ?|)
+					; empty fields for higher levels
+	       hlc (org-minutes-to-hh:mm-string (nth 3 entry)) hlc ; time
+	       "|\n"                                               ; close line
+	       )))))
+      (backward-delete-char 1)
+      (if (setq formula (plist-get params :formula))
+	  (cond
+	   ((eq formula '%)
+	    (setq pcol (+ 3
+			  (if multifile 1 0)
+			  (min maxlevel (or ntcol 100))
+			  (if timestamp 1 0)))
+	    (insert
+	     (format
+	      "\n#+TBLFM: $%d='(org-clock-time%% @%d$%d $%d..$%d);%%.1f"
+	      pcol
+	      (+ 2 (if narrow 1 0))
+	      (+ 3 (if multifile 1 0))
+	      (+ (if multifile 1 0) 3)
+	      (1- pcol)))
+	    (setq recalc t))
+	   ((stringp formula)
+	    (insert "\n#+TBLFM: " formula)
+	    (setq recalc t))
+	   (t (error "invalid formula in clocktable")))
+	;; Should we rescue an old formula?
+	(when (stringp (setq content (plist-get params :content)))
+	  (when (string-match "^\\([ \t]*#\\+TBLFM:.*\\)" content)
+	    (setq recalc t)
+	    (insert "\n" (match-string 1 (plist-get params :content)))
+	    (beginning-of-line 0))))
+      ;; Back to beginning, align the table, recalculate if necessary
+      (goto-char ipos)
+      (skip-chars-forward "^|")
+      (org-table-align)
+      (when org-hide-emphasis-markers
+	;; we need to align a second time
+	(org-table-align))
+      (when recalc
+	(if (eq formula '%)
+	    (save-excursion
+	      (if narrow (beginning-of-line 2))
+	      (org-table-goto-column pcol nil 'force)
+	      (insert "%")))
+	(org-table-recalculate 'all))
+      (when rm-file-column
+	;; The file column is actually not wanted
+	(forward-char 1)
+	(org-table-delete-column))
+      total-time))
+
+(defun org-clocktable-indent-string (level)
+  (if (= level 1)
+      ""
+    (let ((str "\\__"))
+      (while (> level 2)
+	(setq level (1- level)
+	      str (concat str "___")))
+      (concat str " "))))
 
 (defun org-clocktable-steps (params)
+  "Step through the range to make a number of clock tables."
   (let* ((p1 (copy-sequence params))
 	 (ts (plist-get p1 :tstart))
 	 (te (plist-get p1 :tend))
@@ -2008,7 +2117,8 @@ the currently selected interval size."
       (setq p1 (plist-put p1 :tend (format-time-string
 				    (org-time-stamp-format nil t)
 				    (seconds-to-time (setq ts (+ ts step))))))
-      (insert "\n" (if (eq step0 'day) "Daily report: " "Weekly report starting on: ")
+      (insert "\n" (if (eq step0 'day) "Daily report: "
+		     "Weekly report starting on: ")
 	      (plist-get p1 :tstart) "\n")
       (setq step-time (org-dblock-write:clocktable p1))
       (re-search-forward "#\\+END:")
@@ -2016,21 +2126,99 @@ the currently selected interval size."
 	;; Remove the empty table
 	(delete-region (point-at-bol)
 		       (save-excursion
-			 (re-search-backward "^\\(Daily\\|Weekly\\) report" nil t)
+			 (re-search-backward "^\\(Daily\\|Weekly\\) report"
+					     nil t)
 			 (point))))
       (end-of-line 0))))
 
-(defun org-clocktable-add-file (file table)
-  (if table
-      (let ((lines (org-split-string table "\n"))
-	    (ff (file-name-nondirectory file)))
-	(mapconcat 'identity
-		   (mapcar (lambda (x)
-			     (if (string-match org-table-dataline-regexp x)
-				 (concat "|" ff x)
-			       x))
-			   lines)
-		   "\n"))))
+(defun org-clock-get-table-data (file params)
+  "Get the clocktable data for file FILE, with parameters PARAMS.
+FILE is only for identification - this function assumes that
+the correct buffer is current, and that the wanted restriction is
+in place.
+The return value will be a list with the file name and the total
+file time (in minutes) as 1st and 2nd elements.  The third element
+of this list will be a list of headline entries.  Each entry has the
+following structure:
+
+  (LEVEL HEADLINE TIMESTAMP TIME)
+
+LEVEL:     The level of the headline, as an integer.  This will be
+           the reduced leve, so 1,2,3,... even if only odd levels
+           are being used.
+HEADLINE:  The text of the headline.  Depending on PARAMS, this may
+           already be formatted like a link.
+TIMESTAMP: If PARAMS require it, this will be a time stamp found in the
+           entry, any of SCHEDULED, DEADLINE, NORMAL, or first inactive,
+           in this sequence.
+TIME:      The sum of all time spend in this tree, in minutes.  This time
+           will of cause be restricted to the time block and tags match
+           specified in PARAMS."
+  (let* ((maxlevel (or (plist-get params :maxlevel) 3))
+	 (timestamp (plist-get params :timestamp))
+	 (ts (plist-get params :tstart))
+	 (te (plist-get params :tend))
+	 (block (plist-get params :block))
+	 (link (plist-get params :link))
+	 (tags (plist-get params :tags))
+	 (matcher (if tags (cdr (org-make-tags-matcher tags))))
+	 cc range-text st p time level hdl props tsp tbl)
+
+    (setq org-clock-file-total-minutes nil)
+    (when block
+      (setq cc (org-clock-special-range block nil t)
+	    ts (car cc) te (nth 1 cc) range-text (nth 2 cc)))
+    (when (integerp ts) (setq ts (calendar-gregorian-from-absolute ts)))
+    (when (integerp te) (setq te (calendar-gregorian-from-absolute te)))
+    (when (and ts (listp ts))
+      (setq ts (format "%4d-%02d-%02d" (nth 2 ts) (car ts) (nth 1 ts))))
+    (when (and te (listp te))
+      (setq te (format "%4d-%02d-%02d" (nth 2 te) (car te) (nth 1 te))))
+    ;; Now the times are strings we can parse.
+    (if ts (setq ts (org-float-time
+		     (apply 'encode-time (org-parse-time-string ts)))))
+    (if te (setq te (org-float-time
+		     (apply 'encode-time (org-parse-time-string te)))))
+    (save-excursion
+      (org-clock-sum ts te
+		     (unless (null matcher)
+		       (lambda ()
+			 (let ((tags-list (org-get-tags-at)))
+			   (eval matcher)))))
+      (goto-char (point-min))
+      (setq st t)
+      (while (or (and (bobp) (prog1 st (setq st nil))
+		      (get-text-property (point) :org-clock-minutes)
+		      (setq p (point-min)))
+		 (setq p (next-single-property-change
+			  (point) :org-clock-minutes)))
+	(goto-char p)
+	(when (setq time (get-text-property p :org-clock-minutes))
+	  (save-excursion
+	    (beginning-of-line 1)
+	    (when (and (looking-at (org-re "\\(\\*+\\)[ \t]+\\(.*?\\)\\([ \t]+:[[:alnum:]_@#%:]+:\\)?[ \t]*$"))
+		       (setq level (org-reduced-level
+				    (- (match-end 1) (match-beginning 1))))
+		       (<= level maxlevel))
+	      (setq hdl (if (not link)
+			    (match-string 2)
+			  (org-make-link-string
+			   (format "file:%s::%s"
+				   (buffer-file-name)
+				   (save-match-data
+				     (org-make-org-heading-search-string
+				      (match-string 2))))
+			   (match-string 2)))
+		    tsp (when timestamp
+			  (setq props (org-entry-properties (point)))
+			  (or (cdr (assoc "SCHEDULED" props))
+			      (cdr (assoc "DEADLINE" props))
+			      (cdr (assoc "TIMESTAMP" props))
+			      (cdr (assoc "TIMESTAMP_IA" props)))))
+	      (when (> time 0) (push (list level hdl tsp time) tbl))))))
+      (setq tbl (nreverse tbl))
+      (list file org-clock-file-total-minutes tbl))))
+
 
 (defun org-clock-time% (total &rest strings)
   "Compute a time fraction in percent.
@@ -2051,7 +2239,8 @@ This function is made for clock tables."
 	  (if (string-match "\\([0-9]+\\):\\([0-9]+\\)" s)
 	      (throw 'exit
 		     (/ (* 100.0 (+ (string-to-number (match-string 2 s))
-				    (* 60 (string-to-number (match-string 1 s)))))
+				    (* 60 (string-to-number
+					   (match-string 1 s)))))
 			tot))))
 	0))))
 
@@ -2081,7 +2270,8 @@ The details of what will be saved are regulated by the variable
 		   (buffer-file-name b)
 		   (or (not org-clock-persist-query-save)
 		       (y-or-n-p (concat "Save current clock ("
-					 (substring-no-properties org-clock-heading)
+					 (substring-no-properties
+					  org-clock-heading)
 					 ") "))))
 	      (insert "(setq resume-clock '(\""
 		      (buffer-file-name (org-clocking-buffer))
@@ -2162,3 +2352,4 @@ The details of what will be saved are regulated by the variable
 ;; arch-tag: 7b42c5d4-9b36-48be-97c0-66a869daed4c
 
 ;;; org-clock.el ends here
+

lisp/org-exp.el

@@ -1086,6 +1086,9 @@ on this string to produce the exported version."
       ;; Protect short examples marked by a leading colon
       (org-export-protect-colon-examples)
 
+      ;; Protected spaces
+      (org-export-convert-protected-spaces backend)
+
       ;; Normalize footnotes
       (when (plist-get parameters :footnotes)
 	(org-footnote-normalize nil t))
@@ -1536,6 +1539,26 @@ from the buffer."
       (add-text-properties (point) (org-end-of-subtree t)
 			   '(org-protected t)))))
 
+(defun org-export-convert-protected-spaces (backend)
+  "Convert strings like \\____ to protected spaces in all backends."
+  (goto-char (point-min))
+  (while (re-search-forward "\\\\__+" nil t)
+    (org-if-unprotected-1
+     (replace-match
+      (org-add-props
+	  (cond
+	   ((eq backend 'latex)
+	    (format "\\hspace{%dex}" (- (match-end 0) (match-beginning 0))))
+	   ((eq backend 'html)
+	    (org-add-props (match-string 0) nil
+	      'org-whitespace (- (match-end 0) (match-beginning 0))))
+	   ;; ((eq backend 'docbook))
+	   ((eq backend 'ascii)
+	    (org-add-props (match-string 0) '(org-whitespace t)))
+	   (t (make-string (- (match-end 0) (match-beginning 0)) ?\ )))
+	  '(org-protected t))
+      t t))))
+
 (defun org-export-protect-verbatim ()
   "Mark verbatim snippets with the protection property."
   (goto-char (point-min))

lisp/org.el

@@ -18066,6 +18066,23 @@ upon the next fontification round."
       (setq l (- l (get-text-property b 'org-dwidth-n s))))
     l))
 
+(defun org-shorten-string (s maxlength)
+  "Shorten string S so tht it is no longer than MAXLENGTH characters.
+If the string is shorter or has length MAXLENGTH, just return the
+original string.  If it is longer, the functions finds a space in the
+string, breaks this string off at that locations and adds three dots
+as ellipsis.  Including the ellipsis, the string will not be longer
+than MAXLENGTH.  If finding a good breaking point in the string does
+not work, the string is just chopped off in the middle of a word
+if necessary."
+  (if (<= (length s) maxlength)
+      s
+    (let* ((n (max (- maxlength 4) 1))
+	   (re (concat "\\`\\(.\\{1," (int-to-string n) "\\}[^ ]\\)\\([ ]\\|\\'\\)")))
+      (if (string-match re s)
+	  (concat (match-string 1 s) "...")
+	(concat (substring s 0 (max (- maxlength 3) 0)) "...")))))
+
 (defun org-get-indentation (&optional line)
   "Get the indentation of the current line, interpreting tabs.
 When LINE is given, assume it represents a line and compute its indentation."