PATCH: enhances latex filter to cope with forward references, indexes and bibliographies (take 2)

This is the second version of my patch for the Template::Filters module (against the current CVS tree) to enhance the latex filter so that it re-runs latex to resolve forward references, and generate and format the index and/or bibliography if the document includes them. 

This version of the patch includes a check for missing table of contents or list of tables or figures files on the first run, so a document with a table of contents, but no forward references will have latex run at least twice so that the TOC is generated properly (the first run just writes TOC entries to the file doc.toc, the second run reads the file and generates the TOC).  Of course if a document includes a TOC then once the TOC is included in the document the page numbers may change, but the filter should notice the warning that labels have changed and re-run latex to sort out the TOC.

The patch also modifies Makefile.PL and Template::Config.pm to do configuration of the bibtex and makeindex paths.

I wanted to change the Template::Manual::Filters document to reflect the changes I made, but couldn't quite work out where the source document was.  The following text replaces the last paragraph describing the latex filter (if the patch meets with approval, then perhaps someone could apply it and add the text to the documentation source):

The latex filter will re-run latex or pdflatex if the document
contains undefined references after the first run.  It will run bibtex
if it undefined citations are reported, and will run makeindex if it
detects a ".idx" file.  If bibtex or makeindex are run then latex (or
pdflatex) are run up to three more times to allow forward references
to stabilize.

Note that the latex filter runs up to eight external programs, so it
isn't very fast.  But for modest documents the performance is adequate,
even for interactive applications.

I have run a "make test" and everything looks fine.  I have not added tests for forward references, index entries or citations in that I'm not sure how to go reliably poking around in a dvi, ps or pdf file looking for appropriate results.  I have though run ttree over one of my web document hierarchies that uses these features and they do work.  I'll have a bit more of a think about it to get some effective tests written.

I think that with this patch the latex filter will cope with most latex documents.  The main failing is that documents in the source tree cannot be included in the main latex document (with \include{file}) unless one adds the relevant directories to the TEXINPUTS environment variable when ttree is run (or when the Template->process method is invoked).

Andrew

-- 
Andrew Ford,  Director    Pauntley Prints / Ford & Mason Ltd
A.Ford-OfKrLxNBp1iX/4koqx8SDw@xxxxxxxxxxxxxxxx   South Wing Compton House 
pauntley-prints.co.uk     Compton Green, Redmarley  Tel: +44 1531 829900
ford-mason.co.uk          Gloucester GL19 3JB       Fax: +44 1531 829901
refcards.com/cronolog.org Great Britain          Mobile: +44 7785 258278

--- Makefile.PL.orig    2003-10-09 09:42:20.000000000 +0100
+++ Makefile.PL 2004-07-15 12:07:36.000000000 +0100
@@ -16,6 +16,7 @@
 use vars qw( $TT_VERSION $TT_PREFIX $TT_IMAGES $TT_RUN_DBI
             $TT_BUILD_DOCS $TT_SPLASH_DOCS $TT_EXAMPLES $TT_EXTRAS
             $TT_LATEX_ENABLE $TT_LATEX_PATH $TT_PDFLATEX_PATH $TT_DVIPS_PATH 
+            $TT_BIBTEX_PATH $TT_MAKEINDEX_PATH
             $TT_XS_ENABLE $TT_XS_DEFAULT
             $TT_SPLASH_THEME $TT_QUIET $TT_ACCEPT $TT_YES );
 
@@ -76,6 +77,8 @@
 
   TT_LATEX       install LaTeX filter    (y if LaTeX found)
   TT_LATEX_PATH      path to latex       (system dependant)
+  TT_BIBTEX_PATH     path to bibtex      (system dependant)
+  TT_MAKEINDEX_PATH  path to makeindex   (system dependant)
   TT_PDFLATEX_PATH   path to pdflatex    (  "     "     " )
   TT_DVIPS_PATH      path to dvips       (  "     "     " )
 
@@ -109,6 +112,8 @@
 $TT_XS_DEFAULT    = 'y';
 $TT_LATEX_ENABLE  = 'y';
 $TT_LATEX_PATH    = '';
+$TT_BIBTEX_PATH   = '';
+$TT_MAKEINDEX_PATH = '';
 $TT_PDFLATEX_PATH = '';
 $TT_DVIPS_PATH    = '';
 $TT_QUIET         = 'n';
@@ -134,6 +139,8 @@
 $TT_IMAGES        = $ttconfig{ TT_IMAGES     } if $ttconfig{ TT_IMAGES     };
 $TT_SPLASH_THEME  = $ttconfig{ TT_THEME      } if $ttconfig{ TT_THEME      };
 $TT_LATEX_PATH    = $ttconfig{ TT_LATEX_PATH } if $ttconfig{ TT_LATEX_PATH };
+$TT_BIBTEX_PATH   = $ttconfig{ TT_BIBTEX_PATH } if $ttconfig{ TT_BIBTEX_PATH };
+$TT_MAKEINDEX_PATH = $ttconfig{ TT_MAKEINDEX_PATH } if $ttconfig{ 
TT_MAKEINDEX_PATH };
 $TT_PDFLATEX_PATH = $ttconfig{ TT_PDFLATEX_PATH } if $ttconfig{ 
TT_PDFLATEX_PATH };
 $TT_DVIPS_PATH    = $ttconfig{ TT_DVIPS_PATH } if $ttconfig{ TT_DVIPS_PATH };
 $TT_RUN_DBI       = $ttconfig{ TT_DBI        } if defined $ttconfig{ TT_DBI    
    };
@@ -530,6 +537,8 @@
 #    return if $TT_ACCEPT && (! $TT_LATEX_ENABLE || $TT_LATEX_ENABLE eq 'n');
 
     $TT_LATEX_PATH    ||= find_program($ENV{PATH}, "latex")    || '';
+    $TT_BIBTEX_PATH   ||= find_program($ENV{PATH}, "bibtex")    || '';
+    $TT_MAKEINDEX_PATH ||= find_program($ENV{PATH}, "makeindex")    || '';
     $TT_PDFLATEX_PATH ||= find_program($ENV{PATH}, "pdflatex") || '';
     $TT_DVIPS_PATH    ||= find_program($ENV{PATH}, "dvips")    || '';
 
@@ -540,7 +549,7 @@
 -------------
 
 TT2 supports PDF, DVI and PostScript output using the latex filter,
-implemented with the programs pdflatex, latex and dvips.
+implemented with the programs pdflatex, latex, bibtex, makeindex and dvips.
 
 Because the latex filter runs latex and pdflatex, template authors could
 use this feature to include any arbitrary file in their latex input, or
@@ -549,10 +558,12 @@
 your site. If you don't trust your template authors then don't enable
 the latex filter.
 
-I found the following locations for pdflatex, latex and dvips:
-  + pdflatex => $TT_PDFLATEX_PATH
-  + latex    => $TT_LATEX_PATH
-  + dvips    => $TT_DVIPS_PATH
+I found the following locations for pdflatex, latex, bibtex, makeindex and 
dvips:
+  + pdflatex  => $TT_PDFLATEX_PATH
+  + latex     => $TT_LATEX_PATH
+  + bibtex    => $TT_BIBTEX_PATH
+  + makeindex => $TT_MAKEINDEX_PATH
+  + dvips     => $TT_DVIPS_PATH
 
 EOF
 
@@ -567,9 +578,11 @@
                                 $TT_LATEX_ENABLE) =~ /^y/i
                        );
     if ( $TT_LATEX_ENABLE ) {
-        if (ttprompt('Are the pdflatex, latex and dvips paths ok?', 'y') !~ 
/^y/i) {
+        if (ttprompt('Are the pdflatex, latex, bibtex, makeindex and dvips 
paths ok?', 'y') !~ /^y/i) {
             $TT_PDFLATEX_PATH = ttprompt('pdflatex path', $TT_PDFLATEX_PATH);
             $TT_LATEX_PATH    = ttprompt('latex path',    $TT_LATEX_PATH);
+            $TT_BIBTEX_PATH   = ttprompt('bibtex path',    $TT_BIBTEX_PATH);
+            $TT_MAKEINDEX_PATH = ttprompt('makeindex path',    
$TT_MAKEINDEX_PATH);
             $TT_DVIPS_PATH    = ttprompt('dvips path',    $TT_DVIPS_PATH);
         }
     } else {
@@ -581,6 +594,8 @@
     
     fix_file(catfile('lib','Template','Config.pm'), '$PDFLATEX_PATH', 
$TT_PDFLATEX_PATH);
     fix_file(catfile('lib','Template','Config.pm'), '$LATEX_PATH',    
$TT_LATEX_PATH);
+    fix_file(catfile('lib','Template','Config.pm'), '$BIBTEX_PATH',   
$TT_BIBTEX_PATH);
+    fix_file(catfile('lib','Template','Config.pm'), '$MAKEINDEX_PATH', 
$TT_MAKEINDEX_PATH);
     fix_file(catfile('lib','Template','Config.pm'), '$DVIPS_PATH',    
$TT_DVIPS_PATH);
 }
 
@@ -925,6 +940,8 @@
 \$TT_SPLASH_THEME  = '$TT_SPLASH_THEME';
 \$TT_LATEX_ENABLE  = '$ttlatex';
 \$TT_LATEX_PATH    = '$TT_LATEX_PATH';
+\$TT_BIBTEX_PATH   = '$TT_BIBTEX_PATH';
+\$TT_MAKEINDEX_PATH = '$TT_MAKEINDEX_PATH';
 \$TT_PDFLATEX_PATH = '$TT_PDFLATEX_PATH';
 \$TT_DVIPS_PATH    = '$TT_DVIPS_PATH';
 \$TT_XS_ENABLE     = '$ttxs_enable';
--- lib/Template/Config.pm.orig 2004-07-15 12:07:41.000000000 +0100
+++ lib/Template/Config.pm      2004-07-15 12:53:01.000000000 +0100
@@ -29,7 +29,7 @@
 use base qw( Template::Base );
 use vars qw( $VERSION $DEBUG $ERROR $INSTDIR
             $PARSER $PROVIDER $PLUGINS $FILTERS $ITERATOR 
-             $LATEX_PATH $PDFLATEX_PATH $DVIPS_PATH
+             $LATEX_PATH $BIBTEX_PATH $MAKEINDEX_PATH $PDFLATEX_PATH 
$DVIPS_PATH
             $STASH $SERVICE $CONTEXT $CONSTANTS @PRELOAD );
 
 $VERSION   = sprintf("%d.%02d", q$Revision: 2.68 $ =~ /(\d+)\.(\d+)/);
@@ -49,13 +49,15 @@
                $PLUGINS, $PROVIDER, $SERVICE, $STASH );
 
 # the following is set at installation time by the Makefile.PL 
-$INSTDIR  = '';
+$INSTDIR  = '/usr/local/tt2';
 
 # LaTeX executable paths set at installation time by the Makefile.PL
 # Empty strings cause the latex(pdf|dvi|ps) filters to throw an error.
 $LATEX_PATH    = '/usr/bin/latex';
+$BIBTEX_PATH   = '/usr/bin/bibtex';
+$MAKEINDEX_PATH = '/usr/bin/makeindex';
 $PDFLATEX_PATH = '/usr/bin/pdflatex';
-$DVIPS_PATH    = '';
+$DVIPS_PATH    = '/usr/bin/dvips';
 
 #========================================================================
 #                       --- CLASS METHODS ---
@@ -287,14 +289,14 @@
 #------------------------------------------------------------------------
 # latexpaths()
 #
-# Returns a reference to a three element array:
-#    [latex_path,  pdf2latex_path, dvips_path]
+# Returns a reference to a five element array:
+#    [latex_path,  pdflatex_path, dvips_path, bibtex, makeindex]
 # These values are determined by Makefile.PL at installation time
 # and are used by the latex(pdf|dvi|ps) filters.
 #------------------------------------------------------------------------
 
 sub latexpaths {
-    return [$LATEX_PATH, $PDFLATEX_PATH, $DVIPS_PATH];
+    return [$LATEX_PATH, $PDFLATEX_PATH, $DVIPS_PATH, $BIBTEX_PATH, 
$MAKEINDEX_PATH];
 }
 
 #========================================================================
--- lib/Template/Filters.pm.orig        2004-01-30 19:32:25.000000000 +0000
+++ lib/Template/Filters.pm     2004-07-16 09:36:58.000000000 +0100
@@ -599,20 +599,79 @@
 # the entire doc.(pdf|ps|dvi) output or throws an error with a summary
 # of the error messages from doc.log.
 #
+# Re-runs latex (or pdflatex) if there are undefined references.
+# Runs bibtex if the latex log reports undefined citations
+# Runs makeindex if the latex run leaves a ".idx" file in the temporary 
directory.
+# Re-runs latex if bibtex or makeindex was run
+# Re-runs latex up to three more times if labels change.
+#
 # Written by Craig Barratt, Apr 28 2001.
 # Win32 additions by Richard Tietjen.
+# BibTex/MakeIndex/forward reference changes by Andrew Ford.
 #------------------------------------------------------------------------
 use File::Path;
 use File::Spec;
 use Cwd;
 
+sub _run_texish_command
+{
+    my($context, $fName, $currDir, $tmpDir, $command, $logfile) = @_;
+    my $message = "";
+
+    my $fileName = File::Spec->catfile($tmpDir, $logfile);
+    if ( system($command) ) {
+        if ( open(FH, "<$fileName") ) {
+            my $state = 0;
+            #
+            # Try to extract just the interesting errors from
+            # the verbose log file
+            #
+            while ( <FH> ) {
+                #
+                # TeX errors seems to start with a "!" at the
+                # start of the line, and are followed several
+                # lines later by a line designator of the
+                # form "l.nnn" where nnn is the line number.
+                # We make sure we pick up every /^!/ line, and
+                # the first /^l.\d/ line after each /^!/ line.
+                #
+                if ( /^(!.*)/ ) {
+                    $message .= $1 . "\n";
+                    $state = 1;
+                }
+                if ( $state == 1 && /^(l\.\d.*)/ ) {
+                    $message .= $1 . "\n";
+                    $state = 0;
+                }
+            }
+            close(FH);
+        } else {
+            $message = "Unable to open $fileName\n";
+        }
+        my $ok = chdir($currDir);
+        rmtree($tmpDir);
+        $context->throw($fName, "can't chdir $currDir") if ( !$ok );
+        $context->throw($fName, "latex exited with errors:\n$message");
+    }
+    elsif ($logfile) {
+        if ( open(FH, "<$fileName") ) {
+            $message = join('', (<FH>));
+            close(FH);
+        }
+        else {
+            $message = "Unable to open $fileName\n";
+        }
+    }
+    return $message;
+}
+
 sub latex_filter_factory
 {
     my($context, $output) = @_;
 
     $output = lc($output);
     my $fName = "latex";
-    my($LaTeXPath, $PdfLaTeXPath, $DviPSPath)
+    my($LaTeXPath, $PdfLaTeXPath, $DviPSPath, $BibTeXPath, $MakeIndexPath)
                         = @{Template::Config->latexpaths()};
     if ( $output eq "ps" || $output eq "dvi" ) {
         $context->throw($fName,
@@ -637,6 +696,7 @@
             my $cnt = 0;
             my($tmpDir, $fileName, $devnull);
             my $texDoc = 'doc';
+            my ($BibTeX_cmd, $MakeIndex_cmd, $messages);
 
             do {
                 $tmpDir = File::Spec->catdir($tmpRootDir,
@@ -662,48 +722,70 @@
                 $context->throw($fName, "can't chdir $tmpDir");
             }
             #
-            # We don't need to quote the backslashes on windows, but we
-            # do on other OSs
+            # We don't need to quote the backslashes in the LaTeX command line
+            # on windows, but we do on other OSs
             #
             my $LaTeX_arg = "\\nonstopmode\\input{$texDoc}";
             $LaTeX_arg = "'$LaTeX_arg'" if ( $^O ne 'MSWin32' );
-            if ( system("$LaTeXPath $LaTeX_arg"
-                   . " 1>$devnull 2>$devnull 0<$devnull") ) {
-                my $texErrs = "";
-                $fileName = File::Spec->catfile($tmpDir, "$texDoc.log");
-                if ( open(FH, "<$fileName") ) {
-                    my $state = 0;
-                    #
-                    # Try to extract just the interesting errors from
-                    # the verbose log file
-                    #
-                    while ( <FH> ) {
-                        #
-                        # TeX errors seems to start with a "!" at the
-                        # start of the line, and are followed several
-                        # lines later by a line designator of the
-                        # form "l.nnn" where nnn is the line number.
-                        # We make sure we pick up every /^!/ line, and
-                        # the first /^l.\d/ line after each /^!/ line.
-                        #
-                        if ( /^(!.*)/ ) {
-                            $texErrs .= $1 . "\n";
-                            $state = 1;
-                        }
-                        if ( $state == 1 && /^(l\.\d.*)/ ) {
-                            $texErrs .= $1 . "\n";
-                            $state = 0;
-                        }
-                    }
-                    close(FH);
-                } else {
-                    $texErrs = "Unable to open $fileName\n";
+            my $LaTeX_cmd = "$LaTeXPath $LaTeX_arg 1>$devnull 2>$devnull 
0<$devnull";
+
+
+            # Run (pdf)?latex 
+            $messages = _run_texish_command($context, $fName, $currDir, 
$tmpDir, $LaTeX_cmd, "$texDoc.log");
+
+            # If there are undefined (forward) references or a TOC
+            # file was reported as not found then re-run latex
+
+            if ($messages =~ m{ LaTeX \s warning: \s There \s were \s 
undefined \s references
+                                    | No \s file \s $texDoc \. ( toc | lof | 
lot ) }xi) {
+                $messages = _run_texish_command($context, $fName, $currDir, 
$tmpDir, $LaTeX_cmd, "$texDoc.log");
+            }
+
+            # If there are undefined citations then run BibTeX (don't bother 
about the log file)
+            if ($BibTeXPath && $messages =~ /LaTeX warning: Citation \`.*?\' 
on page \d+ undefined/i) {
+                $BibTeX_cmd = "$BibTeXPath $texDoc 1>$devnull 2>$devnull 
0<$devnull";
+                _run_texish_command($context, $fName, $currDir, $tmpDir, 
$BibTeX_cmd);
+            }
+
+            # If latex left behind a ".idx" file then run makeindex
+
+            $fileName = File::Spec->catfile($tmpDir, "$texDoc.idx");
+            if ($MakeIndexPath && -f $fileName) {
+                $MakeIndex_cmd = "$MakeIndexPath $texDoc 1>$devnull 2>$devnull 
0<$devnull";
+                _run_texish_command($context, $fName, $currDir, $tmpDir, 
$MakeIndex_cmd);
+            }
+
+            # If bibtex or makeindex was run then rerun latex
+
+            if ($BibTeX_cmd || $MakeIndex_cmd) {
+                $messages = _run_texish_command($context, $fName, $currDir, 
$tmpDir,
+                                                $LaTeX_cmd, "$texDoc.log");
+
+                # If bibtex was run then latex will report undefined
+                # references on the first run (assuming the
+                # bibliography follows the citations).
+
+                if ($messages =~ /LaTeX warning: There were undefined 
references/i) {
+                    $messages = _run_texish_command($context, $fName, 
$currDir, $tmpDir,
+                                                    $LaTeX_cmd, "$texDoc.log");
+                }
+
+                # After all that the labels may change, especially if
+                # the bibliography or index sections are included in
+                # the table of contents, so latex is run a couple more
+                # times for the labels to settle down.
+
+                if ($messages =~ /LaTeX warning: Label\(s\) may have 
changed/i) {
+                    $messages = _run_texish_command($context, $fName, 
$currDir, $tmpDir,
+                                                    $LaTeX_cmd, "$texDoc.log");
+                }
+                if ($messages =~ /LaTeX warning: Label\(s\) may have 
changed/i) {
+                    $messages = _run_texish_command($context, $fName, 
$currDir, $tmpDir,
+                                                    $LaTeX_cmd, "$texDoc.log");
                 }
-                my $ok = chdir($currDir);
-                rmtree($tmpDir);
-                $context->throw($fName, "can't chdir $currDir") if ( !$ok );
-                $context->throw($fName, "latex exited with errors:\n$texErrs");
             }
+
+
             if ( $output eq "ps" ) {
                 $fileName = File::Spec->catfile($tmpDir, "$texDoc.dvi");
                 if ( system("$DviPSPath $texDoc -o"