NAME

       bup-midx - create a multi-index (.midx) file from several .idx files

SYNOPSIS

       bup midx [-o outfile] <-a|-f|idxnames...>

DESCRIPTION

       bup midx creates a multi-index (.midx) file from one or more git pack index (.idx) files.

       Note: you should no longer need to run this command by hand.  It gets run automatically by
       bup-save(1) and similar commands.

OPTIONS

       -o, --output=filename.midx
              use the given output filename for the .midx file.  Default is auto-generated.

       -a, --auto
              automatically generate new .midx files  for  any  .idx  files  where  it  would  be
              appropriate.

       -f, --force
              force generation of a single new .midx file containing all your .idx files, even if
              other  .midx  files  already  exist.   This  will  result  in  the  fastest  backup
              performance, but may take a long time to run.

       --dir=packdir
              specify the directory containing the .idx/.midx files to work with.  The default is
              $BUP_DIR/objects/pack and $BUP_DIR/indexcache/*.

       --max-files
              maximum number of .idx files to open at a time.  You can use this if  you  have  an
              especially  small  number  of file descriptors available, so that midx can complete
              (though possibly non-optimally) even if it can’t open all your .idx files at  once.
              The default value of this option should be fine for most people.

       --check
              validate  a  .midx  file  by  ensuring that all objects in its contained .idx files
              exist inside the .midx.  May be useful for debugging.

EXAMPLES

              $ bup midx -a
              Merging 21 indexes (2278559 objects).
              Table size: 524288 (17 bits)
              Reading indexes: 100.00% (2278559/2278559), done.
              midx-b66d7c9afc4396187218f2936a87b865cf342672.midx

DISCUSSION

       By default, bup uses git-formatted pack files, which consist of a  pack  file  (containing
       objects)  and  an  idx file (containing a sorted list of object names and their offsets in
       the .pack file).

       Normal idx files are convenient because it means you can use git(1) to access your  backup
       datasets.   However, idx files can get slow when you have a lot of very large packs (which
       git typically doesn’t have, but bup often does).

       bup .midx files consist of a single sorted list of all the objects contained  in  all  the
       .pack files it references.  This list can be binary searched in about log2(m) steps, where
       m is the total number of objects.

       To further speed up the search, midx files also have a variable-sized  fanout  table  that
       reduces  the  first n steps of the binary search.  With the help of this fanout table, bup
       can narrow down which page of the midx file a given object id would be in (if  it  exists)
       with a single lookup.  Thus, typical searches will only need to swap in two pages: one for
       the fanout table, and one for the object id.

       midx files are most useful when creating new backups, since searching  for  a  nonexistent
       object  in  the  repository  necessarily requires searching through all the index files to
       ensure that it does not exist.  (Searching for objects that do exist can be optimized; for
       example,  consecutive objects are often stored in the same pack, so we can search that one
       first using an MRU algorithm.)

SEE ALSO

       bup-save(1), bup-margin(1), bup-memtest(1)

BUP

       Part of the bup(1) suite.

AUTHORS

       Avery Pennarun <apenwarr@gmail.com>.