NAME
makepp_scanning -- How makepp finds include files and other hidden dependencies
DESCRIPTION
Makepp can guess additional dependencies or targets for certain commands that it knows something about. This is especially important for C/C++ compilation, where it is too error-prone to list manually all of the include files that a given source file depends on. By looking at the compilation command and the source files themselves, makepp is able to determine accurately which object files need to be rebuilt when some include file changes.
Makepp looks at the first word of your command line. If it recognizes it, it uses the scanner corresponding to that first word. Specifically, it isolates the first word and looks it up in its table; if nothing is found, it strips off the directory information and looks it up again. (This means that you can specify the path to your compiler and makepp will still recognize it.) Currently, makepp recognizes most C/C++ compiler names. It also recognizes commands invoking sh and libtool; for these commands, it skips to the first word in the command that's not an option and looks it up in the table again.
If makepp thinks it's compiling a C/C++ source but can't find a scanner, it will give a warning message to let you know. This usually means that you buried your compiler command too deeply in the action for makepp to find it. For example, I have seen rules like this:
%.o: %.c
@echo Compiling $< now; obscure_wrapper gcc -c $< $(CFLAGS) -o $@The first words of the actions here are echo and obscure_wrapper, for which therer are no scanners, so makepp will not scan for include files in this case. You can ignore the prefixed command by:
register-scanner obscure_wrapper skip-wordThe following sections document the built in scanners. In the name you can use - interchangeably with _.
C/C++ compilation (c-compilation, gcc-compilation)
The C/C++ scanner, which handles both languages indifferently, but with a special variant for gcc's many options, is activated by a command beginning with a compile program that makepp knows about.
It looks at the command for -Idir options specifying the include path or -Ldir options specifying the link path. It then scans any source files for #include directives, and also looks at the command line to see if there are any source files or libraries mentioned which are not listed as dependencies. It recognizes these by their extension.
This scanner gives a warning message if files included with #include "file.h" are not found, or not buildable by makepp, in the include path, or in the directory containing the file which is #including, or in /usr/include. No warning is given if a file included with #include <file.h> is not found. Makepp assumes it is in some system include directory that the compiler knows about, and that files in system include directories won't change.
In addition, files in /usr/include, /usr/local/include, /usr/X11R6/include, and any other directory which is not writable are not scanned to see what they include. Makepp assumes that these files won't change. (If you're running as root, the writability test is performed with the UID and GID of the directory you ran makepp from. This is so compiling a program as an ordinary user and then doing makepp install as root won't cause extra directories to be scanned.)
This is a fairly simple-minded scanner. It will get confused if you do things like this:
#ifdef INCLUDE_THIS
#include "this.h"
#endifbecause it doesn't know about preprocessor conditionals. This is usually harmless; it might cause additional extra files to be labelled as dependencies (occasionally causing unnecessary rebuilds), or else it might cause makepp to warn that the include file was not found. You can either ignore the warning messages, or put an empty file this.h out there to shut makepp up.
If your compiler has a funny name, you can say either of
register-scanner obscure_c_compiler c-compilation
register-scanner obscure_gcc_alias gcc-compilationEmbedded SQL C/C++ compilation (esqlc-compilation)
These commands, which come with the various databases, preprocess special sections in otherwise C/C++-like sources, and produce C/C++ headers and sources. This finds EXEC SQL INCLUDE "filename" or $INCLUDE "filename" directives.
If your preprocessor is not recognized, you can say
register-scanner obscure_esqlc_preprocessor esqlc-compilationSwig (swig)
Swig (Simplified Wrapper and Interface Generator, http://www.swig.org) is a program that converts a C/C++ header file into the wrapper functions needed to make your code callable from a variety of other languages, such as perl, python, tcl, C#, ruby, ocaml, and probably some others that I don't know about.
Makepp understands and parses the swig command line, looking for -I and -l options. It also knows how to scan swig's interface definition files (.i files) looking for %include, %import, and also #include if -includeall is in effect.
If your swig has a funny name, you can say
register-scanner obscure_swig_alias swigOther special command words (skip-word)
Makepp recognizes the following command words and many more and skips over them appropriately in in its search for the correct scanner: condor_compile, distcc, ignore_error, libtool, noecho purify, sh.
If you have more such commands, you can say
register-scanner command skip-wordLibtool
Libtool is a very clever compilation system that greatly simplifies making shared libraries by hiding all the system-dependent details away in a shell script. The only difficulty is that the library binary files are not actually stored in the same directory as the output file--libtool actuall creates a subdirectory, .libs, which contains the real files. This is ordinarily not a problem, but makepp has to know where the real binaries are if it is to link them in from a repository. At the moment, libtool libraries (.la files) are not linked in from repositories; they are always rebuilt if needed. Also, makepp at the moment is not able to use the dependency information that is stored inside the .la file itself. This will hopefully change soon.
Suppressing the scan (none)
Sometimes you may not want a rule or a certain command to be scanned. You can turn off scanning with
register-scanner cc noneQuickscan and smartscan
The :quickscan and :smartscan rule options, if applicable, affect the way that files are scanned.
In :quickscan mode (the default), all include directives are assumed active. This allows for very efficient scanning.
In :smartscan mode, an attempt is made to interpret macros and expressions so that inactive include directives are ignored. For example, the executable produced by compiling the following C program ought not to depend on foo.h:
#if 0
#include "foo.h"
#endif
int main() { return 0; }In the future, :smartscan might become the default.
Custom scanners
Writing your own scanner is somewhat involved, but it is possible. There are 3 ways to hook into this:
If scanner foo is specified in a rule option, and there is a subroutine called parser_foo defined in the makefile's package, then that subroutine must return an object of type
Mpp::Lexer, which will be used to determine a command parser for every command.If scanner foo is specified in a rule option, and there is a subroutine called scanner_foo defined in the makefile's package, then that subroutine must either do all of the scanning work (which is deprecated), or return an object of type
Mpp::CommandParserwhich will be used to parse every command.If no scanner is specified in a rule option, then an object of the
Mpp::Lexerbase class is used. That object determines the scanner based on the first word of each command. The base name is sought as a key in %scanners hash in the makefile's package, which is seeded with all of the default scanners. This hash can be extended using the register_scanner and register_command_parser statements. The value from this hash lookup is a coderef that either does the scanning work (which is deprecated), or returns an object of typeMpp::CommandParser.
In most cases, objects of type Mpp::CommandParser should instantiate at least one object of type Mpp::Scanner. The Mpp::Scanner base class takes care of the distinction between quickscan and smartscan. Note that the behavior of Mpp::Scanner can be markedly affected by this distinction, but that should be transparent to the derived class if it is well-formed. New derived Mpp::Scanner classes ought to be tested in both modes.
If you write your own Mpp::Scanner class, you should also base your rescanning decision on the build info RESCAN. This gets set by makeppreplay after signing files without scanning. So despite the signatures being consistent, a rescan is still necessary. If your Mpp::Scanner uses the inherited scan_file1 method, you're probably fine.
If you want to specify a particular Mpp::CommandParser class for a rule, then you can specify a scanner with a rule option, and arrange for the parser_* subroutine to return an object of type Mpp::ActionParser::Specific.
For more details, refer to the respective class documentation. For examples, see Mpp::CommandParser::Gcc and Mpp::CommandParser::Vcs.
Caching scanner info
If the all of the scanner's important side effects are effected through calls to methods of the Mpp::CommandParser base class, then those side effects can be cached in the build info file, so that they can be played back by a subsequent invocation of makepp without doing all of the costly scanning work. This can save quite a bit of time, especially in smartscan mode.
If the scanner has other important side effects, then it should call the Rule object's mark_scaninfo_uncacheable method. Otherwise, the scanner info retrieved from the build info may be inaccurate, causing the build result possibly to be incorrect. This method is called automatically when a value from the %scanners hash does not return an object of type Mpp::CommandParser, or when the scanner is specified with a rule option, no parser_* subroutine is defined and the scanner_* routine does not return an object of type Mpp::CommandParser.
Cached scan info is invalidated using criteria similar to those used for determining when the target is out of date. Similarly, it can be retrieved from a repository using criteria similar to those used for determining when a target can be linked in from a repository.
You can force makepp to ignore the cached scanner info with the --force-rescan option. This is useful when a broken scanner may have caused incorrect scanner info to be cached.
Ad Hoc Scanner
Often you will have just one or few files which contain dependency information. You don't want to write this into a makefile reduntantly (since redundancy later often leads to inconsistencies when one update gets forgotten). But you also don't want to write a Mpp::Scanner? As a workaround you can generate an include file on the fly. For example Qt has .qrc files which can look like:
<RCC>
<qresource prefix="...">
<file>abc</file>
<file>xyz</file>
...If you adhere to the above layout, you can transform the relevant lines into a makepp include file, which gets automatically created by being included.
%.qrc.makepp: %.qrc
&grep 's!<RCC>\n!$(stem).cc:! || s! *<file>! ! && s!</file>\n!!' $(input) -o $(output)
include $(wildcard *.qrc) # .makepp is appended automatically