Provided by: jigdo-file_0.7.3-3ubuntu3_amd64 
NAME
jigdo-file - Prepare files for Jigsaw Download (distribution of huge files, e.g. CD
images).
SYNOPSIS
jigdo-file COMMAND
[ --image=cdrom.iso ] [ --jigdo=cdrom.jigdo ] [ --template=cdrom.template ] [ --force ] [
MORE OPTIONS ] [ FILES ... | --files-from=f ]
Common COMMANDs: make-template, make-image, verify
DESCRIPTION
Jigsaw Download, or short jigdo, is a scheme developed primarily to make it easy to
distribute huge filesystem images (e.g. CD (ISO9660) or DVD (UDF) images) over the
internet, but it could also be used for other data which is awkward to handle due to its
size, like audio/video files or large software packages.
jigdo tries to ensure that the large file (always called image from now on) is downloaded
in small parts which can be stored on different servers. People who want to download the
image do so by telling the jigdo(1) (NOT IMPLEMENTED YET) download tool to process one
`.jigdo' file; using it, jigdo downloads the parts and reassembles the image. jigdo-file
is used to prepare the files for download.
What makes jigdo special is that the parts that are used to reconstruct the image can have
any size and content - they only need to be contained in a contiguous region anywhere in
the image.
For example, if you wish to distribute an ISO9660 image which contains a snapshot of an
FTP server, you can instruct jigdo-file to prepare the download data in such a way that
when people use jigdo to download the image, jigdo actually fetches the individual files
from the FTP server and assembles them into an exact copy of your image - during the
download! (If the image is not a filesystem dump, you can use split(1) to create the small
parts that the image will be reassembled from.)
You are completely free to choose where the individual parts of the image are stored: They
may be in entirely different directories on different servers (e.g. because of
storage/bandwidth constraints), but this is invisible to the people downloading your
image. The information about available servers only needs to be added to the `.jigdo' file
by you before distributing it.
The `DETAILS' section below contains technical details on how jigdo works. The `EXAMPLES'
section lists a number of common scenarios and may help you to get an idea of what jigdo
is useful for.
OPTIONS
Many options are specific to a particular COMMAND; the ones below are general or used by
several commands. Further options are listed below with the individual commands. All
options are silently ignored if they are not applicable to the current command. For any
BYTES parameters to options, you can append one of the letters `k', `M' or `G' to the
amount you specify, to indicate kilobytes, megabytes or gigabytes.
-h --help
Output short summary of commands and options.
-H --help-all
Output complete summary of commands and options.
-v --version
Output program version.
-i --image=cdrom.iso
Specify location of the file containing the image. The image is the large file that
you want to distribute.
-j --jigdo=cdrom.jigdo
Specify location of the Jigsaw Download description file. The jigdo file is a
human-readable file generated by jigdo-file, to which you add information about all
the servers you are going to upload the files to. jigdo will download this file as
the first step of retrieving the image.
-t --template=cdrom.template
Specify location of the image `template' file. The template file is a binary file
generated by jigdo-file, it contains information on how to reassemble the image and
also (in compressed form) all the data from the image which was not found in any of
the parts.
Depending on the command, each of these three files is used sometimes for input,
sometimes for output. If the file is to be used for output for a particular command
and the output file already exists, jigdo-file exits with an error, unless --force
is present.
In most cases, you will only need to specify one out of -i -j -t, because any
missing filenames will be deduced from the one you specify. This is done by first
stripping any extension from the supplied name and then appending nothing (if
deducing --image), `.jigdo' or `.template'.
-r --report=default|noprogress|quiet|grep
Control how verbose the program is, and what format the output has: noprogress is
the same as default except that no `x% done' progress messages are printed. quiet
restricts the output to what is absolutely necessary, mostly error messages. grep
is only different from default for the make-template command: It enables output in
a simple `<offset> <file>' format which is useful when searching for binary files
in other binary files.
-f --force
Overwrite existent output files without complaining.
--no-force
This is the default. Refuse to overwrite existent output files.
-c --cache=jigdo-cache.db
jigdo-file usually needs to read the entire contents of all the FILES you specify.
If you use it repeatedly (e.g. because you make a new CD image available daily),
caching the file information will increase the program's speed significantly. The
cache file is automatically created if it is not yet present. Data is usually both
read from and written to it.
--no-cache
This is the default. Do not use a cache.
--cache-expiry=SECONDS
Set maximum age of cache entries. Any entries older than this will be removed from
the cache. The default is 30 days. You can append one of the letters `h', `d', `w',
`m', `y' to denote hours, days, weeks, months or years, respectively. A value of
`0' or `off' disables expiry, so that all entries will stay in the cache forever.
See the section `CACHE FILES' below for more information.
--readbuffer=BYTES
Set size of internal buffers. The default is 128k - if you have a fast disc,
increasing this value may make jigdo-file faster, but in general, changing it is
not necessary.
--md5-block-size=BYTES
Uninteresting internal parameter. Set size of blocks into which files are
subdivided. The default is 128k. If you change it, any cache file will have to be
regenerated. Internally, jigdo-file may choose to use a slightly larger or smaller
value.
-T --files-from=file
Read file and directory names from the specified file. If file is `-', read names
from standard input. Each line in the file is taken as a name, so the names may
contain spaces, but not newline characters. An empty line causes jigdo-file to stop
reading from the file.
find(1) is a powerful tool for generating file lists, but make sure to use `find
-type f' if possible - otherwise, if you instruct find to output both a filename
and a symlink to that filename, jigdo-file will read the file contents twice.
--hex Output checksums in hexadecimal instead of Base64-like format. This should not be
used with the make-template command, because the resulting `.jigdo' file violates
the `.jigdo' file format. Its intended use is to make jigdo-file more interoperable
with other Unix shell utilities like md5sum(1).
--no-hex
This is the default. Use jigdo's own Base64-like encoding of checksums.
--debug[=help|=all|=UNIT,~UNIT... ]
Switch on or off debugging output. Just `--debug' is equivalent to `--debug=all'.
The argument is a comma-separated list of unit names for which debugging output is
to be enabled, or disabled if the name is preceded by `~'. The special name `all'
means all units. By default, debugging output is switched off except for the units
`assert' and `general'. The exact list of available units for which debugging can
be switched on depends on whether jigdo was compiled with debugging support - the
list can be printed with `--debug=help'.
FILES Names of files or directories to use as input. These are the parts that are
contained in the image. In case one of the names is a directory, the program
recursively scans the directory and adds all files contained in it. While doing
this, it follows symbolic links, but avoids symlink loops.
If one of the filenames starts with the character `-', you must precede the list of
files with `--'. A value of `-' has no special meaning in this list, it stands for
a file whose name is a single hyphen.
COMMANDS
The command name is the first non-option argument passed to jigdo-file. Most commands have
short abbreviations as well as long names. The short command names should not be used in
scripts - there may be incompatible changes to them in the future!
MAKE-TEMPLATE, MT
Reads image and FILES, creates `.jigdo' and `.template'. This is the main functionality of
jigdo-file.
It is possible to specify both --image=- and --files-from=-. In this case, first the list
of files is read from standard input until an empty line is encountered. Everything
following it is assumed to be the image data. This can be useful if you use mkisofs(1) or
similar programs that can output the complete image on their standard output, because
there is no need to store the image on disc temporarily.
If a FILES argument contains the characters `//' (Unix) or `\.\' (Windows), this has
special meaning. In the final jigdo file that users will download, each of the parts is
referenced in the `[Parts]' section with a URI of the form `Label:some/filename'. (See
`FORMAT OF .JIGDO FILES' below for a detailed description.) The `[Servers]' section gives
a mapping of labels to servers on the internet, with lines like
`Label=http://myserver.org/jigdofiles/'. Using this information, jigdo will create the
final download URI for the part, `http://myserver.org/jigdofiles/some/filename'.
Specifying `//' (or `\.\') in a file or directory name serves to `cut off' the names at
the right directory level. For example, if the Unix path of one of your FILES is
`/path/some/filename', you can tell jigdo-file to cut off after the `/path' by passing it
the argument `/path//some/filename', or `/path//' if you want the whole directory scanned.
The path names need not be absolute; `somedirectory//' is also possible.
--label Label=/path
Specify a name to use as the label name for a path on disc. (Influences the output
jigdo file.) If you used `//' in the FILES arguments as described above, jigdo-file
will by default pick label names automatically (`A', `B' etc.). With this option,
you can give labels more meaningful names. Note that the label name will only be
used if one or more FILES begin with `/path//'.
Try to use label names that start with uppercase characters, to disambiguate them
clearly from protocol names like `http', `ftp'.
--uri Label=http://some.server.org/
By default, using --label as described above will cause lines of the form
`Label=file:/path/' to be written to the `[Servers]' section of the output jigdo
file. If you want to override the `file:' URI so that the line reads
`Label=http://some.server.org/', you can do so by specifying --uri along with
--label. Giving just --uri Label=... without the corresponding --label Label=...
has no effect, and even if you specify both, an entry is only added to the
`[Servers]' section if the label is referenced by at least one `[Parts]' entry.
The supplied value is not quoted by the program; if it contains characters such as
space or any of the characters #"'\ then you must quote it. (Under Unix, you may
need to quote the value twice to also protect it from the shell, e.g. \\\\ or '\\'
to get a single backslash in the URI.)
The mapping specified with an --uri option is ignored if it is already present in
the output jigdo file.
Users of the Windows version may notice that the `\' directory separators are
converted into `/' in the `file:' URIs that are generated by default. This is done
to increase cross-platform compatibility of `file:' - the print-missing command of
the Windows version will automatically re-convert the characters when it prints the
URIs. In case you supply your own `file:' URIs under Windows using --uri, you must
also exchange `/' and `\'.
-0 to -9
Set amount of compression in the output template file, from -0 (no compression) to
-9 (maximum compression). The default is -9, which can make the template generation
quite slow. By default, the compression algorithm used is the same as for gzip(1).
--gzip and --bzip2
Choose between the gzip and bzip2 compression algorithms. The default is gzip.
Bzip2 usually gives a better compression ratio, but compression is significantly
slower than with gzip.
--min-length=BYTES
Set minimum length of a part for jigdo-file to look for it in the image. The
default is 1k. Parts smaller than this will never be found in the image, so their
data will be included in the template file. The search algorithm used requires such
a minimum length, otherwise template generation could become extremely slow. If you
know for sure that all your FILES are larger than a certain amount, you can
increase jigdo-file's speed slightly by specifying the amount with this option.
There is a hard-wired absolute minimum of 256 bytes - anything lower will silently
be set to 256.
--merge=FILE
Include the contents of FILE in the output `.jigdo' file. The file can contain data
which you want added to the output (for example, a `[Servers]' section with a list
of your servers as entries), or it can be the jigdo file output by an earlier run
of jigdo-file.
It is possible to specify the same file for input with --merge and for output with
--jigdo. However, you will also need to use --force to make the program overwrite
the old version of the jigdo file with the new one. FILE can be `-' for standard
input.
When adding new information to the supplied file, jigdo-file will not insert new
lines into the `[Parts]' section if an entry for the same MD5 checksum (but not
necessarily with the same URI!) already exists, and it will not insert new lines
into the `[Servers]' section if a completely identical entry already exists.
When reading in the existing FILE, the behaviour is slightly different: The program
preserves entries in the `[Parts]' section with identical checksum, but different
URIs. For completely identical entries (same checksum and URI), only one entry is
preserved and the duplicates are removed. The `[Servers]' section is left
untouched.
--image-section
This is the default. Causes jigdo-file to add an `[Image]' section to the `.jigdo'
file.
As an exception, a new `[Image]' section is not added if you use --merge and the
file to merge contains an `[Image]' section with a line which reads `Template-
MD5Sum=' (end of line after the `='). In this case, the generated template data's
MD5 checksum value is just added after the `=' of the first line of this form in
the file - no whole new `[Image]' section is appended. This behaviour is useful
because it allows you to pass via --merge an `[Image]' section with arbitrary
content and then have the MD5 checksum automatically added by jigdo-file. The
section `FORMAT OF .JIGDO FILES' below explains the `[Image]' section contents in
greater detail.
--no-image-section
Do not include an `[Image]' section in the `.jigdo' file. You need to add one
yourself if you use this option. However, doing that is not easy (you also need to
add a `Template-MD5Sum' line with the correct checksum, or jigdo will complain), so
use of this option is discouraged.
--servers-section
This is the default. Causes jigdo-file to add a `[Servers]' section to the `.jigdo'
file. This default section uses `file:' URIs, which allows for immediate reassembly
of the image from the local filesystem, and is also useful if you want to edit the
file manually and replace the `file:' URIs with other URIs.
--no-servers-section
Do not add a `[Servers]' section at the end of the `.jigdo' file. Useful e.g. if
you are going to append the section with a script.
--match-exec=SHELLCOMMAND
Whenever a file is found in the image, execute the supplied command string by
passing it to a shell. jigdo-file sets up a number of environment variables with
information about the file match. For example, if the file `/path//a/b/file' was
found in the image and `Label:a/b/file' is going to be written to the `.jigdo'
file:
• LABEL="Label" - Name of the label for the file. The example assumes that `--label
Label=/path' was specified by you. In the absence of such an option, LABEL will
be set but empty.
• LABELPATH="/path/" - The path corresponding to the label, or in other words, the
prefix of the matched file's path that will not appear in the output `.jigdo'
file. Is set even without any `--label' option present. Ends with a slash.
• MATCHPATH="a/b/" - The rest of the path, without the leafname of the matched
file. Is either empty or ends with a slash.
• LEAF="file" - The leafname of the matched file.
• MD5SUM="lNVdUSqbo2yqm33webrhnw" - The md5sum of the matched file, in Base64-like
format.
• FILE="/path//a/b/file" - For convenience, the complete path of the file. The
variable is always set to $LABELPATH$MATCHPATH$LEAF.
Please be careful to correctly quote the string passed to this option, otherwise your
supplied command will not work with filenames that contain spaces. As an example, to
create a backup of hard links to the matched files, use the following option: --match-
exec='mkdir -p "${LABEL:-.}/$MATCHPATH" && ln -f "$FILE" "${LABEL:-.}/$MATCHPATH$LEAF"'
By default, no command is executed. Use --match-exec="" to remove a command string which
was set with an earlier use of this option.
--greedy-matching
This is the default. Imagine that your image contains a .tar file which in turn
contains another file x, and that you provide both the .tar and the files inside it
on the command line. When jigdo-file scans the image, it encounters the beginning
of the .tar file, and then the file x.
At this point, a decision must be made: Should the smaller file x be recorded as
matched, or should it be ignored in favour of the larger (and thus better) match of
the .tar file? Unfortunately, at this point it is not clear whether there will
actually be a full match of the .tar, so by default, the program prefers the small
match.
--no-greedy-matching
In the case where a large partial match is present and a shorter match has been
confirmed, ignore the small match. (See the option above.)
MAKE-IMAGE, MI
Reads `.template' and FILES, creates image (or `imagename.tmp'). Provides a rudimentary
way of reassembling images - jigdo is usually better suited for this task. However, in
contrast to jigdo, no `.jigdo' file is required.
If the image is to be written to a file (and not to standard output), it is possible to
create the image in several steps, with several invocations of `jigdo-file make-image', as
follows: You first invoke jigdo-file, specifying as many files as are available at this
time. The program scans the files, and those that are contained in the image are copied to
a temporary file, whose name is formed by appending `.tmp' to the image filename.
For all further files which could be parts of the image, you repeat this process. As soon
as all parts are present, the temporary file will be truncated slightly (to delete some
administrative data that jigdo-file appends at the end) and renamed to the final image
name. The possibility of reassembling the image in several steps is especially useful for
gathering files from removable media, e.g. several older CDs.
Scripts using make-image can detect whether image creation is complete by checking the
exit status: 0 signals successful creation, whereas 1 means that more files need to be
supplied. Other errors result in an exit status of 2 (`recoverable', e.g. file not found)
or 3 (non-recoverable, e.g. write error).
--check-files
This is the default. Whenever any part is copied to the image, re-check its
checksum against the checksum stored in the template. It is recommended that you
leave this switched on, even if it slows down image creation a bit.
--no-check-files
Do not check files' checksums when copying them to the image. This can be safely
used when no cache file is used (which means that files will be written to the
image immediately after being scanned) or the whole image is checked later with the
verify command.
PRINT-MISSING, PM
Reads `.jigdo', `.template' and (if present) `imagename.tmp', outputs a list of URIs still
needed to completely reassemble the image.
Together with the make-image command, this provides most of the functionality of jigdo on
the command line.
For each part that is not yet present in the temporary image file, the file checksum is
looked up in the `[Parts]' section of the jigdo file. Any label in the corresponding entry
is then expanded according to the label definitions in the `[Servers]' section and printed
on standard output. jigdo allows you to specify several alternative locations for each
label in this section, but print-missing will only output the first one for each missing
part.
If the checksum cannot be found in the `[Parts]' section (this Should Not Happen unless
you deleted that section), a lookup is instead made for `MD5Sum:<checksum>', just like
with jigdo. (Thus, if you want to get rid of the `[Parts]' section, you can do so if you
rename each part to its own checksum.)
--uri Label=http://some.server.org/
Override the entries in the `.jigdo' file for any label with a URI of your choice.
With the example above, a `[Parts]' entry of `Label:some/filename' will cause the
line `http://some.server.org/some/filename' to be printed.
The supplied value is not quoted by the program; if it contains characters such as
space or any of the characters #"'\ then you must quote it. (Under Unix, you may
need to quote the value twice to also protect it from the shell, e.g. \\\\ or '\\'
to get a single backslash in the URI.)
PRINT-MISSING-ALL, PMA
Just like print-missing, this command outputs a list of URIs still needed to completely
reassemble the image. However, all alternative download locations are printed instead of
just one. In the output, the URIs for a file are separated from other files' URIs with
blank lines. The --uri option has the same effect as for print-missing.
VERIFY, VER
Reads image (presumably generated with make-image) and `.template', checks for correct
checksum of image.
The template data does not only contain checksums of the individual parts, but also of the
image as a whole. make-image already performs a number of internal checks, but if you
like, you can additionally check the image with this command.
SCAN, SC
Reads all the FILES and enters them into the cache, unless they are already cached. The
--cache option must be present for this command.
--no-scan-whole-file
This is the default. This only causes the first --md5-block-size bytes of each file
to be read. If the cache is used later by jigdo-file make-image, the rest of the
file will be read once these first bytes are recognized in the input image.
--scan-whole-file
Immediately read the entire file contents and store them in the cache.
MD5SUM, MD5
Reads all the FILES and prints out MD5 checksums of their contents. This command is quite
similar to md5sum(1), except that the checksum is output in the Base64-like encoding which
is also used elsewhere by jigdo-file.
The FILES arguments are processed in the same way as with the other commands, which means
that recursion automatically takes place for any arguments that are directories, and that
symbolic links are not listed except when the file(s) they point to are not reachable
directly.
In the checksum list printed on standard output, only the part of the filename following
any `//' (or `\.\' on Windows) is printed. Any --cache will be used for querying files'
MD5 checksums and/or writing the checksums of scanned files.
LIST-TEMPLATE, LS
Reads a `.template' file and outputs low-level information about the image and all parts
contained in it, including offset, length and checksum.
You can also use this command with temporary image files (by specifying something like
--template=imagename.tmp) - in that case, the output also distinguishes between parts that
have been written to the image and parts that haven't.
The exact output format may change incompatibly between different jigdo releases. The
following different types of lines can be output. `have-file' only occurs for `.tmp'
files, indicating a file that has already been successfully written to the temporary file:
in-template offset-in-image length
need-file offset-in-image length file-md5sum filestart-rsyncsum
have-file offset-in-image length file-md5sum filestart-rsyncsum
image-info image-length image-md5sum rsyncsum-size
DETAILS
Jigsaw Download was created with the format of ISO9660 CD images in mind - however, the
following also applies to many other filesystem formats, as well as to `tar' archives and
uncompressed `zip' archives. A CD image contains both information for organizing the
filesystem (header with disc name etc., ISO9660 directory data, data of extensions such as
Joliet or RockRidge, zero padding) and the files contained on the CD. An important
property that jigdo relies on is that each file is stored in one contiguous section of the
image; it is not split into two or more parts.
When jigdo-file is given a number of files that might be contained in an image, it detects
whether any of the files are present using a `rolling checksum' inspired by the one used
by rsync(1). The resulting data is written to the `.template' file: If a section of the
image could not be matched (e.g. it was directory information), the data is compressed and
written directly to the template. However, if a matching file was found, its data is
omitted from the template. Instead, only a reference (an MD5 checksum of the file) is
inserted in the template.
Note that the template data only contains binary data, it does not contain any filenames
or URIs, since it cannot be easily edited in case any of these values need to be changed.
All that information is stored in the `.jigdo' file, a text file to which you can add URLs
for your server(s). The jigdo file provides a mapping for each MD5 checksum to one or more
alternative download locations for the corresponding part.
Apart from the mapping of MD5 sums to URIs, the jigdo file also contains an URI pointing
to a download location for the template file. This way, the jigdo download tool only needs
to be given one URI (that of the `.jigdo' file) to be able to download and reassemble the
complete image.
FORMAT OF .JIGDO FILES
The overall format of `.jigdo' files follows that of `.ini' files, as also used by the
Gnome and KDE projects for some data. The file is organized into sections, each of which
is preceded by a line reading `[Sectionname]'. Within each section, lines have the form
`Label=Value'. Such lines are also called `entries' below. All `.jigdo' files use UTF-8 as
their character encoding.
Comments are introduced with the `#' character and extend to the end of the line.
Whitespace is ignored at line start and end as well as to the left and right of section
names and the `=' in entries. Furthermore, the jigdo utilities split up the text of the
entry value (i.e. the part after the `=') into whitespace-separated words, much like the
Unix shell. Single '' and double "" quotes can be used to prevent that e.g. URIs
containing whitespace are split apart. Similarly, characters with special meaning (the
characters '"#\ and space/tab) must be quoted with \ to appear in the value. As with the
shell, there is a difference between ' ' and " ": Within ' ', the characters "#\ and
whitespace lose their special meaning and become ordinary characters, whereas within " ",
only the characters '# and whitespace lose their special meaning - in other words,
backslash escapes still work inside " ", but not ' '.
`.jigdo' files can optionally be compressed with gzip(1). jigdo-file always outputs
uncompressed files, which you can compress yourself. jigdo-lite supports single
uncompressed and compressed files.
(Behaviour which may change in the future and which should not be relied upon: jigdo
additionally supports any number of concatenated plaintext and gzipped parts in the files
- for example, you can compress a `.jigdo' file and then add a couple of lines of
uncompressed data to the end.)
In all cases, the `.gz' extension should be removed from the filename - the tools will
determine automatically from the file contents whether a file is compressed or not.
Below is a description of the individual section names used by jigdo.
JIGDO SECTION
[Jigdo]
Version=1.1
Generator=jigdo-file/1.0.0
Information about the version of the jigdo file format used, and the program that
generated it. There should be one such section per `.jigdo' file.
IMAGE SECTION
[Image]
Filename="filename for saving on user's disc"
Template="URI where to fetch template file"
Template-MD5Sum=OQ8riqT1BuyzsrT9964A7g
ShortInfo=single-line description of the image (200 characters max.)
Info=long description (5000 characters max.)
The value for the `Template' entry can be either an URL (absolute or relative to the URL
of the jigdo file) or a string of the form `Label:pathname' (UNIMPLEMENTED), as described
below.
The `Template-MD5Sum' entry is added by jigdo-file and specifies the MD5 checksum of the
generated `.template' file. It is used by jigdo to detect cases where the downloaded
template data is corrupted or belongs to a different image.
Unlike other entry values, the values of the `ShortInfo' and `Info' entries are not split
up into words, instead all quoting is preserved.
The value of the `Info' entry is special in that jigdo(1) can optionally parse XML markup
it contains. If the markup has errors such as unbalanced/unsupported tags, the string is
displayed literally, without XML parsing. Supported tags are <b></b> (bold), <i></i>
(italic), <tt></tt> (typewriter font), <u></u> (underline), <big></big> (larger font),
<small></small> (smaller font) and <br/> (linebreak). Supported entities include <
(`<'), > (`>') and & (`&'). Note that the whole `Info' entry must be on one line in
the jigdo file.
This section may occur multiple times, but all except the first one will be ignored. This
is useful e.g. when creating a `.jigdo' file for a DVD image when you already have
`.jigdo' files for CDs with the same content: You can simply `[Include]' (see below) the
CDs' jigdo files at the end of the DVD jigdo file, after its `[Image]' section.
PARTS SECTION
[Parts]
xJNkjrq8NYMraeGavUpllw=LabelA:part0
GoTResP2EC6Lb_2wTsqOoQ=LabelA:part1
kyfebwu6clbYqqWUdFIyaw=LabelB:some/path/part2
-J9UAimo0Bqg9c0oOXI1mQ=http://some.where.com/part3
All lines in the section, which provides the mapping from MD5 checksums to URIs, have the
same format: On the left side of the `=' the checksum (encoded with a Base64-like
encoding) is given, and on the right a string corresponding to the part with this
checksum; either a complete URI or a string of the form `Label:pathname', which is
expanded into one or more URIs by looking up the definition(s) for the Label in the
`[Servers]' section.
In case a particular MD5 checksum cannot be found in any `[Parts]' section by jigdo, the
program will perform a lookup for `MD5Sum:<checksum>', e.g. for
`MD5Sum:xJNkjrq8NYMraeGavUpllw' if you deleted the line for `part0' above.
A checksum appearing multiple times in this section indicates alternative download
locations for the part.
There may be any number of `[Parts]' sections in the file; they are all considered when
looking up MD5 checksums.
jigdo-file always puts the `[Parts]' section at the end of the file, and it even
rearranges any file specified with --merge to have only one such section at the end. This
is done to allow jigdo to display the information from the `[Image]' section while the
rest of that file is still being downloaded.
SERVERS SECTION
[Servers]
LabelA=http://myserver.org/
LabelA=ftp://mirror.myserver.org/
LabelB=LabelC:subdirectory/
LabelC=http://some.where.com/jigdo/
All lines in the section, which provides the mapping from server labels to server
locations, have the same format: On the left side of the `=' the label name is given, and
on the right the value to expand the label name to.
A label name appearing multiple times in this section indicates alternative download
locations for the parts that use the label in the `[Parts]' section. This notation makes
it very easy to add mirrors to the jigdo file.
As shown by the example above, the label values may themselves reference other labels. In
this case, the entry `LabelB:some/path/part2' in the `[Parts]' section will expand to
`http://some.where.com/jigdo/subdirectory/some/path/part2'. Loops in the label
definitions result in undefined behaviour and must be avoided.
There may be any number of `[Servers]' sections in the file; they are all considered when
looking up labels. Either of `[Parts]' or `[Servers]', but not both, can be omitted from
the jigdo file.
COMMENT SECTION
[Comment]
Any text, except that lines must not begin with `['.
All text following a `[Comment]' or `[comment]' line is ignored, up to the next line with
a section label.
INCLUDE DIRECTIVE
[Include http://some.url/file.jigdo]
Lines of this form cause the content of the specified jigdo file to be downloaded and
parsed just like the main jigdo file. The effect will be the same as copying the included
file's contents into the file which contains the include directive. (Exception: Any
relative URLs are always resolved using the URL of the `.jigdo' file that contains that
relative URL.)
The URL argument can be an absolute or relative URL. Relative URLs are assumed to be
relative to the URL of the jigdo file which contains the include directive. Includes can
be nested, but it is an error to create a loop of include directives. It is not possible
to use URLs of the form `Label:pathname'.
The URL cannot be quoted with "". Any `]' characters in the argument must be escaped as
`%5D', and any spaces as `%20'.
Include directives are only supported by jigdo, they are ignored by jigdo-lite.
An include directive terminates any previous section, but it does not start a new one. In
other words, a new section must always be started after the include line, jigdo does not
allow normal entries to appear below the `[Include]'.
CACHE FILES
Any file specified with the --cache option is used to store information about the FILES
presented to jigdo-file. When querying the cache, a file is considered unchanged (and the
cached data is used) only if filename, file size and last modification time (mtime) match
exactly. For the filename match, not the entire file name is used, but only the part
following any `//', so that any changes to the part before the `//' will not invalidate
the cache.
Old cache entries are removed from the cache if they have not been read from or written to
for the amount of time specified with --cache-expiry. Entries are not immediately removed
from the cache if the file they refer to no longer exists - this makes it possible to
cache information about files on removable media.
Cache expiry only takes place after jigdo-file has done its main work - if any old entries
are accessed before expiry takes place, they will be kept. For example, if the program is
run using the default expiry time of 30 days, but accesses a cache file with entries
generated 2 months ago, then entries in that cache will be considered, and only those
cache entries that were not needed during the program run will be expired.
Due to a peculiarity of the underlying database library (libdb3), cache files never
shrink, they only grow. If a large number of entries was expired from your cache file and
you want it to shrink, you can either just delete it (of course then everything will have
to be regenerated) or use the utilities accompanying libdb3 to dump and restore the
database, with a command like `db3_dump old-cache.db | db3_load new-cache.db'. For Debian,
these programs are supplied in the package `libdb3-util'.
If a different --md5-block-size is specified, the entire file needs to be re-read to
update its cache entry. If a different --min-length is specified, only the first
`md5-block-size' bytes of the file need to be re-read.
EXAMPLES
PREPARING YOUR CD IMAGE FOR DISTRIBUTION
You have created a CD image `image.iso' from some of the files stored in the directory
`/home/ftp' on your harddisc, which is also available online as `ftp://mysite.org'. As
you don't want to waste space by effectively hosting the same data twice (once as files on
the FTP server, once inside the image), and you are fed up with users' downloads aborting
after 200MB and their restarting the download dozens of times, you decide to use jigdo.
How do you prepare the image for download?
In fact, only one command is necessary:
jigdo-file make-template --image=image.iso --jigdo=/home/ftp/image.jigdo
--template=/home/ftp/image.template /home/ftp// --label Mysite=/home/ftp --uri
Mysite=ftp://mysite.org/
People can now point jigdo at `ftp://mysite.org/image.jigdo' to download your image. The
template file needs to be accessible as `ftp://mysite.org/image.template'.
Note that nothing prevents you from doing the same for an FTP server that isn't
administrated by you - in that case, you only need to host the `.jigdo' and `.template'
files on your own server/homepage.
PREPARING AN ARBITRARY LARGE FILE FOR DISTRIBUTION
We assume that you have a large file that is not a filesystem, e.g. `movie.mpeg'. Because
of space problems, you want to distribute the data on two servers.
In this case, the parts of the image need to be generated artificially with the split
command. For example, to create chunks of 4MB each, use `split -b 4m movie.mpeg part'.
Copy the resulting files `partXX' into two directories `1' and `2' that you create,
according to how you want the files distributed between the servers. Next, create the
jigdo and template files with `jigdo-file make-template --image=movie.mpeg 1// 2//'. You
will need to edit the `.jigdo' file and provide the right URIs for the two servers that
you are going to upload the `partXX' files to.
CUSTOMIZED VERSIONS OF IMAGES
Because it is possible to assign a different URI for each part of an image if necessary,
jigdo is very flexible. Only one example is the possibility of customized versions of
images: Suppose that someone is distributing a CD image, and that you want to make a few
small changes to it and redistribute your own version. You download the `official.iso' CD
image with jigdo (passing it the URL of `official.jigdo'), write it to CD-R, make your
changes (say, adding files from the `myfiles' directory on your harddisc) and produce your
own version, `myversion.iso'. Next, you instruct jigdo-file to create the jigdo and
template files for your modified image, using the command
jigdo-file make-template --image=myversion.iso /mnt/cdrom/ myfiles// --label
My=myfiles/ --uri My=http://my.homepage.net/ --merge=official.jigdo
while `official.iso' is mounted under `/mnt/cdrom'. By using --merge, you have told jigdo-
file to take the contents of `official.jigdo', add to it a new `[Image]' section for
`myversion.iso' and write the resulting jigdo file to `myversion.jigdo' - so now
`myversion.jigdo' offers two images for download, the original version and your modified
version. (If you do not want it to offer the official version, edit it and remove the
`[Image]' section that lists `official.iso'.)
Now you can upload the `.jigdo' file, the `.template' file and also the files in `myfiles'
to `http://my.homepage.net/'. Thus, for people to download your modified image, you do
not need to upload the complete image contents to your web space, but only the changes you
made!
(In case you only made very few changes, you could also omit the `myfiles' parameter in
the command above, then all your changes end up in the new template file.)
COMBINING MANY JIGDO-MANAGED IMAGES INTO ONE
It is also no problem to combine data from several sources that use jigdo. For example, if
of five different and unrelated servers each one distributes a different CD image via
jigdo, you can create a customized DVD image that contains the data from all these CDs.
When people use jigdo to download your image, the individual files on the DVD are fetched
from the same sources as the original CDs.
Consequently, even though you will be distributing a 3.2GB file via your web space, the
actual amount of data that is stored on your server will only be in the order of several
MBs.
BUGS
For certain contents of one of the input files, most notably a sequence of zero bytes
longer than --min-length at the start of the file and an area of zeros preceding the file
data in the image, jigdo-file make-template may fail to find the file in the image.
Unfortunately, this restriction cannot be avoided because the program could become very
slow otherwise. If you use the --debug option, all instances of jigdo-file discarding
possible matches are indicated by lines containing the word `DROPPED'.
In fact, not only all-zeroes files trigger this behaviour, but also files which contain at
their start a long sequence of short identical strings. For example, both a file
containing only `a' characters and one containing `abcabcabcabc...' are problematic.
SEE ALSO
jigdo(1) (NOT YET IMPLEMENTED), jigdo-lite(1), jigdo-mirror(1), split(1) (or `info
split'), find(1) (or `info find'), mkisofs(1), md5sum(1)
AUTHOR
Jigsaw Download <URL:http://atterer.net/jigdo/> was written by Richard Atterer <jigdo
atterer.net>, to make downloading of CD ROM images for the Debian Linux distribution more
convenient.
19 May 2006 JIGDO-FILE(1)