NAME

       makepp_repositories -- How to use repositories for variant builds, for maintaining a
       central set of sources, and other things

DESCRIPTION

       A repository is a directory or directory hierarchy outside of the default directory that
       contains files which the makefile needs in the current directory tree.  Makepp can
       automatically link files from the repository into the current directory tree if they are
       needed.  Repositories provide similar functionality to the "VPATH" variable, but (unlike
       "VPATH" in other versions of make) you do not have to do anything special to your makefile
       to get them to work.

       Repositories are specified with the "-R" or "--repository" command line option or with the
       "repository" statement in the makefile.  Note that if you have a habit of calling makepp
       in different subdirectories of your build tree, it is easy to accidentally reimport a
       repository somewhere else.  As a safeguard against this, if you use RootMakeppfile, makepp
       will refuse to start if it finds one above or below where it would be imported.

       This is somewhat comparable to operating system union filesystems (unionfs...)  The
       current directory is like the highest level writable layer.  All repositories are like
       lower read-only layers.

       Repositories are useful in several different situations:

       •   When you want to place your object and executable files in a separate directory, but
           the makefile is written to place them in the same directory as the sources.

       •   When you want to build the same program two different ways (e.g., with two different
           sets of compilation options, or for two different architectures).

       •   When you don't have write access to all or part of the source tree.

       •   When several developers are working on the same project, and there is a common source
           repository containing all the sources for the project.  Each developer can modify only
           the files he needs to change in his local directory without affecting the other
           developers, and makepp will automatically fetch the unmodified files from the source
           repository.

       Makepp's implementation of repositories does not require rewriting of the build commands
       at all, unlike (for example) repositories in cons.  Makepp puts a symbolic link into the
       directory where the command is expecting it.  As long as the command does not refer to
       absolute directories, the exact same shell command will work with files from a repository.
       This means that it works not only for compilation commands, but any kind of command you
       can think to put in your makefile.

       Makepp has another kind of mechanism called a build cache which solves some of the same
       sorts of problems as repositories in a different way.  Depending on your problem, a build
       cache may be more useful than a repository.  See makepp_build_cache for information about
       build caches and a comparison of build caches and repositories.

   Examples
       Repositories are best explained by several examples of what you can do.

       Different compilation options

       Suppose you have a simple program with a makefile that looks something like this:

           CFLAGS      = -O2
           OBJECTS = a.o b.o c.o
           my_program: $(OBJECTS)
               cc $(inputs) -o $(output)

           %.o: %.c
               cc $(CFLAGS) -c $(input) -o $(output)

       This makefile places the files "a.o", "b.o", "c.o", and "my_program" in the same directory
       as the source files.

       Sometimes you want to place the binary files into a separate directory.  For example, you
       might build your program on several different architectures, and you don't want the binary
       files on one architecture to be replaced with the binary files on the other.  Or you might
       want to make a temporary change and recompile without wiping out the previous compilation
       results.  Without repositories, you would have to modify your makefile to place the
       objects elsewhere.

       With a repository, however, you don't have to touch your makefile at all.  Consider the
       following sequence of commands:

           % cd my_program_source
           % makepp                    # Builds using the above makefile, and
                                       # object files go into the directory
                                       # my_program_source.
           % cd ..
           % mkdir binary-debug        # Make a clean directory for building the
           % cd binary-debug           # same program with different options.
           % makepp -R ../my_program_source CFLAGS=-g
                                       # Now objects go into binary-debug.

       The first makepp command compiles the source files with optimization and puts the objects
       into the directory "my_program_source", because that's what the makefile is supposed to
       do.  Now we want to rebuild the program, but we want to change the value of "CFLAGS" to
       compile for debug.  We specify the new value of "CFLAGS" on the command line, and we also
       tell makepp that the "my_program_source" directory is a repository using the "-R" option.

       Every time makepp realizes that it needs a file that it doesn't already have in current
       directory, it looks in the repository.  In this case, it first looks for the makefile,
       which doesn't exist in the "binary-debug" subdirectory.  So it creates a symbolic link to
       it from the makefile in "my_program_source", and then reads in the makefile.  Then it
       notices that it needs the file "a.c" in order to build "a.o", and so it links in "a.c"
       from the repository.  If "a.c" includes any files contained in "my_program_source", then
       these will be automatically linked in as well.  Note: Those links are useful for things
       like debugging, but if you don't like them, "makeppclean -R" can remove them.

       Running the build command in "binary-debug" won't touch any of the files in
       "my_program_source".  Thus from the same set of source files, you now have two different
       copies of the program, one compiled with optimization and one compiled for debug.  And
       this happened without touching the makefile at all.

       The advantage of using repositories instead of simply recompiling and overwriting the
       original binaries is that now if we fix our bugs and want to go back to the optimized
       version, we don't have to recompile everything.  Since the original object files are still
       around, and most of them are still valid, we can save a lot of time on recompilation.
       This does not make a big difference when only three source files are involved, but for a
       larger build that takes minutes or hours to complete, the savings in programmer time and
       frustration can be significant.

       Rebuilding one file with a minor modification to the compilation commands

       Makepp doesn't fetch only source files from the repository.  If the object files in the
       repository don't need rebuilding, it will use them.  For example, consider a slight
       modification to the above makefile:

           CFLAGS   := -O2
           A_CFLAGS := -O6 -funroll-loops

           OBJECTS := a.o b.o c.o

           my_program: $(OBJECTS)
               cc $(inputs) -o $(output)

           %.o: %.c
               cc $(CFLAGS) -c $(input) -o $(output)

           a.o: a.c
               cc $(A_CFLAGS) -c $(input) -o $(output)

       The idea is that "a.o" contains the time-critical code, so it is compiled with higher
       optimization than the rest of the objects.  Now suppose we want to test just how different
       the timing is with different compile options.  A repository can help with this, too:

           % cd my_program_source
           % makepp                    # Builds using the above makefile, and
                                       # object files go into the directory
                                       # my_program_source.
           % cd ..
           % mkdir no-unrolling        # Make a clean directory for building the
           % cd no-unrolling           # same program with different options.
           % makepp -R ../my_program_source A_CFLAGS=-O2
           % cd ..
           % time no-unrolling/my_program # Benchmark the two versions of the program.
           % time my_program_source/my_program

       Makepp proceeds as before, linking in a copy of the makefile and then examining the object
       files.  Now only the "a.o" module needs recompiling, since the options for "b.o" and "c.o"
       haven't changed.  Makepp notices that it can use "b.o" and "c.o" from the repository, so
       it just links those in.  However, it will recompile "a.o" in the "no-unrolling" directory.
       Once the compilation is finished, the two different versions of the program can be
       benchmarked.

       Rebuilding with a minor modification to the source

       Now suppose we want to make a change to "a.c" and benchmark the program before and after
       the change.  Repositories can help again.  Consider this sequence of commands:

           % mkdir modified-a
           % cp my_program_source/a.c modified-a
           % cd modified-a
           % emacs a.c                 # Make some modifications just to this module.
           % makepp -R ../my_program_source

       Here we have created a new directory that just contains the single source file we want to
       modify.  Makepp now takes "a.c" from the "modified-a" subdirectory, but uses the copies of
       "b" and "c" from the "my_program_source" directory.  Without changing any of the binary
       files in "my_program_source", we have created a separate copy of the program that
       incorporates our changes to "a.c".  If there are other developers using the sources in
       "my_program_source", they will be unaffected by our changes.

       Repositories can thus be used as a quick way to build variants of a program, without
       adding complicated conditions to the makefile.  None of the files in the original
       directory are modified; they are used as needed.

       Using a directory hierarchy

       A repository is actually not just a single directory, it's a whole directory hierarchy.
       Suppose you use /our/library as a repository.  Now /our/library may well contain many
       subdirectories, e.g., /our/library/gui and /our/library/network.  Consider this command:

           % makepp -R /our/library

       Any commands in the makefile that refer to files in the directory ./network will actually
       get files from /our/library/network, and similarly for ./gui.  Makepp automatically
       creates any directories that exist in the repository but not in the current directory.

       Linking to any place in the file system

       All of the above examples show files from a repository being linked into the current
       directory or its subdirectories, but you can actually have makepp link them into any place
       in the file system that you have write access to.  This is done by specifying
       "-R new-location=old-location".

       For example, sometimes it's a little tedious to type the following:

              mkdir alternate-build
              cd alternate-build
              makepp -R ..

       You can do it all with one command, like this:

              makepp -R alternate-build=. -F alternate-build

       "-F" or "-makeppfile" changes to that directory before loading the makefile.  You must
       specify "-R" before "-F".  Note that this example puts the new build tree inside the
       repository.  That will not work if you use a RootMakeppfile because makepp safeguards
       against nested trees.  It's also not a good idea if you use **, because if you ever build
       in the repository it will also find edited and generated files in this subtree.

       Assigning a different location in the file system may be also useful for more complicated
       builds, where there are several library subdirectories.  For example, here's a command I
       have used to build variants of one of my programs:

           % makepp -R test-build/seescape=/src/seescape \
                -R test-build/HLib=/src/HLib \
                -R test-build/H5pp=/src/H5pp \
                -R qwt=/src/external_libraries/qwt \
                -F test-build/seescape

       This command loads in files from four different repositories, and then cds to the
       ./test-build/seescape directory and executes the makefile there.  Files contained in the
       directory tree beginning with /src/seescape are linked into ./test-build/seescape.  In
       other words, makepp will temporarily link the file /src/seescape/gui/image_canvas.cxx to
       ./test-build/seescape/gui/image_canvas.cxx when it is needed.  This command will work even
       if the "test-build" directory doesn't exist yet; makepp will create it for you.  (But you
       must specify the "-R" options before the "-F" option on the command line.)

       Multiple equivalent repositories

       Say your project is maintained by several fairly autonomous groups.  You could have one
       complete repository with all the sources as they are in production or at least
       successfully tested.  Every group can have a mostly empty repository with (part of) the
       same structure, containing the files group members have finished developing.

       Developers' current directories will have the files they are still working on.  The group
       repository will be the first one given and the production repository the last one, so that
       it furnishes the files not found in the group repository:

           $ makepp -R/path/to/group/repository -R/path/to/production/repository

       Since this is probably fairly static for that directory, you may want to put a file
       .makepprc at its root with the following content:

           -R/path/to/group/repository -R/path/to/production/repository

       Or, presuming that it has a fixed path, you could write into your makefile:

           repository /path/to/production/repository

       and, because options are seen before makefiles are read, you can then call just

           $ makepp -R/path/to/group/repository

       Repositories as fixed part of your build system

       If you know you always use some repository you can use the "repository" or "vpath"
       statements in your makefile.

   Caveats with repositories
       When the links get in the way

       For finding your way around your file hierarchy and for allowing the debugger to find the
       sources it is useful to have the links used while building.  But when you want to edit a
       file or resync it with your version control, the links can get in the way.  That is
       because the system traverses the link and writes to the file in the repository.  Unless
       it's your personal repository used just for keeping things apart, that may not be what you
       want.

       As a safeguard against inadvertent overwriting of public files it is suggested to make the
       sources in the repository unwritable.  It might even not be enough to remove the write
       bit, because a version control system which insists on your locking the files for editing
       might also do that, but temporarily make the file writable while resyncing it.  If that is
       the case for you, the repository should actually belong to a different user.

       There are a few tactics to surmount this:

       •   Keep the sources you edit in a repository, separate from your build tree.  Whenever
           you put a file, which was previously fetched from another repository, into this
           editing repository, makepp will notice and fetch it from there, provided it is the
           first repository you specify.

       •   Remember to delete any file, before you create a copy for writing.  If you follow the
           safeguard suggestion above, forgetting to do this will give an error message when
           writing.  To help you, the following function "delink" will replace one link by a copy
           of the linked file.  The first variant is for all kinds of Bournish Shells, the second
           one for csh (or at least tcsh):

               $ delink() { { rm $1 && cat >$1; } <$1; }
               % alias delink '( rm \!:1 && cat >\!:1; ) <\!:1'

       •   If you feel you don't need them, you can delete them all, whenever you want, e.g.
           after every makepp run, possibly backgrounded (either short or long form):

               makeppclean --recurse --only-repository-links
               mppc -rR

       Don't build in a repository during use

       A repository is meant to be read-only while it is being used as a repository.  Makepp will
       not work properly if you change files in your repository during the course of a build.
       Nightly builds may be ok for you, if no one else uses the repository at that time.  Before
       it starts the build, makepp gets a list of all the files that exist in the repository, and
       never updates its list, except for files it expects to appear.

       If you need a repository that's changing as you build, you might want to consider makepp's
       build cache mechanism (see makepp_build_cache).  Alternatively, you can use a "poor man's
       repository": you can put explicit rules into your makefile to create the soft links, like
       this:

           %.c : $(directory_I_wish_was_a_repository)/%.c
               &ln -fs $(input) $(output)

       This works only for source files; you can't easily use this to link a file if it is
       already built in the repository, but build it here if it's not already built, since there
       is only allowed to be one way to build a file.

       Use only relative filenames

       Repositories work completely transparently if the makefiles use only relative filenames.
       In the above example, it's ok if the makefile in /src/seescape refers to ../HLib, but the
       above command will not work as expected if it refers to /src/HLib.  If you need to use
       absolute file names, you can put them into make variables and then override them on the
       command line, like this:

           % makepp -R test-build/seescape=/src/seescape SEESCAPE=/home/holt/test-build/seescape \
                -R test-build/HLib=/src/HLib HLIB=/home/holt/test-build/HLib \
                -R test-build/H5pp=/src/H5pp H5pp=/home/holt/test-build/H5pp \
                -R qwt=/src/external_libraries/qwt QWT=/home/holt/test-build/qwt \
                -F test-build/seescape

       The above will work as long as the "HLib" directory is referred to as "$(HLIB)" in all the
       makefiles.  Note that you have to specify absolute paths for the directories, because
       makepp cd's to "test-build/seescape" before reading the makefile.  This leads to long and
       complicated make commands; use relative paths when possible.

       Makepp must know about all dependencies

       Repositories will not work if there are hidden dependencies that makepp doesn't know
       about.  (In fact, doing a build using repositories, is one way of checking for forgotten
       dependencies.  But, just for this check, don't combine it with a build cache, since
       fetching something there, instead of building it, might hide a forgotten dependency.)
       Sometimes these dependencies can be fairly subtle.  For example, the libtool command will
       not only create ".lo" and ".la" files as listed on the command line, but it also may
       create a subdirectory called ".libs" which contains the actual object files.  To prevent
       build mistakes, makepp refuses to link in a ".la" file from a repository.  Hopefully in
       the future libtool will be better supported.

       Many hidden dependencies related to compilation are caught by the command line scanner.
       If your compiler uses the common Unix compilation flags (e.g., "-I", "-D", etc.), then
       makepp will usually figure out where all your include files are.  You may have to be
       careful if you have any homegrown scripts that create files that makepp doesn't know
       about.  For correct builds, it is vitally important to list all targets and dependencies
       (or determine them automatically by scanning).

       Putting absolute filenames into programs

       Repositories will also not work if any of the files built contain absolute file names in
       them (e.g., if any of your build commands write out an absolute filename).  For example,
       it turns out that the ".la" files produced by libtool have this property.  (If you look at
       the contents of the ".la" file you'll see that the dependency list contains absolute
       filenames.)  In order to solve this particular problem, makepp will not link ".la" files
       from a repository; it will insist on rebuilding them.

       Avoid linking in unnecessary directories

       Repositories can be slow on startup and use a lot of memory if there are a lot of
       unnecessary files in the repository.  For example, if you use an automatic HTML
       documentation generator which makes thousands of ".html" files from your source code, you
       may not want to put them in a subdirectory of a directory that's used as a repository.
       It's better to put them in a different directory tree entirely, so the repository
       mechanism won't load in their names.

       Too Many Files

       The disadvantage of repositories is that symbolic links, which the repository mechanism
       uses, are individual files (though they use almost no disk space).  This is unlike real
       links, but those can't cross file system boundaries.  In extreme cases the presence of
       very many symbolic links can lead to exhaustion of the number of foreseen files (so called
       inodes), even though there is plenty of space left.  In this case the sysadmin will need
       to tune the file system.

   Overriding repository copies
       If you make any modifications to a file locally, makepp will ordinarily realize this and
       recompile the file using the local copy rather than the repository copy.

       If you're using a repository to maintain a central code base, and you have developers
       working on local copies which contain only the files they have modified, one problem that
       comes up is: what if a developer wants to remove a file from his local build but the
       repository still contains it?  If the developer removes the local copy, makepp will
       happily put in the copy from the repository, and the build will proceed as if the file
       existed.

       One technique (alas not for user root) for this problem is to make the file that you want
       not to include in the build process unreadable, like this:

           chmod a-rw file-to-be-excluded

       This will prevent makepp from incorporating it from the repository.  Makepp also includes
       special code so that unreadable files do not match wildcards or pattern rules.

       Similarly, to prevent makepp from incorporating an entire subdirectory, make a local
       directory that has the same name but is unwritable.  If you want makepp to ignore the
       directory entirely, then make it unreadable too.  (Read-only directories are searched but
       targets in them are usually not built.)

       The other way to do this is calling makepp with one or more exclusion options:

           mpp -R /path/to/rep --dont-read=/path/to/rep/file-to-be-excluded

   Don't use repositories for files which can change!
       Don't try to use a repository for a file which is part of your build.  For example, you
       might be tempted to try to use repositories to put all of your public .h files in the same
       directory, like this:

           # top level makefile
           repository include=module1/include
           repository include=module2/include
           repository include=module3/include
           repository include=module4/include

       This is probably not a good idea if any of the .h files are themselves outputs of a
       program (e.g., yacc or some other program that spits out C source code), because makepp
       assumes that files in repositories never change.  If the build needs include/xyz.h, and
       module2/include/xyz.h actually needs to be produced by some program, makepp will not know
       to run the program.  It's better to use a technique like this to put all of your .h files
       into a common include directory:

           # module1/Makeppfile
           ../include/%.h : include/%.h
               &cp $(input) $(output)
           # You could also (more efficiently but problematic on Windows) do the following:
           #    &ln -r $(input) $(output)

       Makepp might still try to build files that happen to be in a repository if something asks
       for them directly, but it won't build them on behalf of the local directory.  The result
       of this can be quite confusing, because it can lead to a repository symbolic link being
       used while its repository target is out-of-date, but that target might get updated later
       in the build.  You can prevent this from happening either by making sure that the
       repository is referred to only through the repository path, or by making sure that there
       is also a local rule for all the generated repository files.

       Another way to avoid recompiling identical files in different directories is to use a
       build cache (see makepp_build_cache for details).  A build cache does not have the
       restriction that the file may not change.