Provided by: obnam_1.6.1-1_amd64 
NAME
obnam - make, restore, and manipulate backups
SYNOPSIS
obnam [--checkpoint=SIZE] [--chunk-size=SIZE] [--chunkids-per-group=NUM] [--client-name=CLIENT-NAME]
[--compress-with=PROGRAM] [--config=FILE] [--crash-limit=COUNTER] [--critical-age=AGE]
[--deduplicate=MODE] [--dump-config] [--dump-memory-profile=METHOD] [--dump-setting-names]
[--encrypt-with=ENCRYPT-WITH] [--exclude=EXCLUDE] [--exclude-caches] [--fsck-fix] [--fsck-ignore-chunks]
[--fsck-ignore-client=NAME] [--fsck-last-generation-only] [--fsck-skip-dirs] [--fsck-skip-files]
[--fsck-skip-generations] [--fsck-skip-per-client-b-trees] [--fsck-skip-shared-b-trees] [--fuse-opt=FUSE]
[--generate-manpage=TEMPLATE] [--generation=GENERATION] [-h] [--help] [--help-all]
[--idpath-bits=IDPATH-BITS] [--idpath-depth=IDPATH-DEPTH] [--idpath-skip=IDPATH-SKIP] [--keep=KEEP]
[--key-details] [--keyid=KEYID] [--leave-checkpoints] [--list-config-files] [--lock-timeout=TIMEOUT]
[--log=FILE] [--log-keep=N] [--log-level=LEVEL] [--log-max=SIZE] [--log-mode=MODE] [--lru-size=SIZE]
[--memory-dump-interval=SECONDS] [--no-default-configs] [--no-exclude-caches] [--no-fsck-fix]
[--no-fsck-ignore-chunks] [--no-fsck-last-generation-only] [--no-fsck-skip-dirs] [--no-fsck-skip-files]
[--no-fsck-skip-generations] [--no-fsck-skip-per-client-b-trees] [--no-fsck-skip-shared-b-trees]
[--no-key-details] [--no-leave-checkpoints] [--no-one-file-system] [--no-pretend] [--no-dry-run]
[--no-no-act] [--no-pure-paramiko] [--no-quiet] [--no-small-files-in-btree] [--no-strict-ssh-host-keys]
[--no-verbose] [--no-weak-random] [--node-size=SIZE] [--one-file-system] [--output=FILE] [--pretend]
[--dry-run] [--no-act] [--pretend-time=TIMESTAMP] [--pure-paramiko] [--quiet] [-r=REPOSITORY]
[--repository=REPOSITORY] [--root=ROOT] [--sftp-delay=SFTP-DELAY] [--small-files-in-btree]
[--ssh-command=EXECUTABLE] [--ssh-host-keys-check=VALUE] [--ssh-key=FILENAME]
[--ssh-known-hosts=FILENAME] [--strict-ssh-host-keys] [--symmetric-key-bits=SYMMETRIC-KEY-BITS]
[--testing-fail-matching=REGEXP] [--to=TO] [--trace=TRACE] [--upload-queue-size=SIZE] [--verbose]
[--verify-randomly=N] [--version] [--viewmode=MODE] [--warn-age=AGE] [--weak-random]
obnam [options] add-key [CLIENT-NAME]...
obnam [options] backup [DIRECTORY]...
obnam [options] client-keys
obnam [options] clients
obnam [options] convert5to6
obnam [options] diff [GENERATION1]GENERATION2
obnam [options] force-lock
obnam [options] forget [GENERATION]...
obnam [options] fsck
obnam [options] generations
obnam [options] genids
obnam [options] help
obnam [options] help-all
obnam [options] list-keys
obnam [options] list-toplevels
obnam [options] ls [FILE]...
obnam [options] mount [ROOT]
obnam [options] nagios-last-backup-age
obnam [options] remove-client [CLIENT-NAME]...
obnam [options] remove-key [CLIENT-NAME]...
obnam [options] restore [DIRECTORY]...
obnam [options] verify [DIRECTORY]...
DESCRIPTION
obnam makes, restores, manipulates, and otherwise deals with backups. It can store backups on a local
disk or to a server via sftp. Every backup generation looks like a fresh snapshot, but is really incre‐
mental: the user does not need to worry whether it's a full backup or not. Only changed data is backed
up, and if a chunk of data is already backed up in another file, that data is re-used.
The place where backed up data is placed is called the backup repository. A repository may be, for exam‐
ple, a directory on an sftp server, or a directory on a USB hard disk. A single repository may contain
backups from several clients. Their data will intermingle as if they were using separate repositories,
but if one client backs up a file, the others may re-use the data.
obnam command line syntax consists of a command possibly followed by arguments. The commands are list
below.
• backup makes a new backup. The first time it is run, it makes a full backup, after that an incre‐
mental one.
• restore is the opposite of a backup. It copies backed up data from the backup repository to a
target directory. You can restore everything in a generation, or just selected files.
• clients lists the clients that are backed up to the repository.
• generations lists every backup generation for a given client, plus some metadata about the genera‐
tion.
• genids lists the identifier for every backup generation for a given client. No other information
is shown. This can be useful for scripting.
• ls lists the contents of a given generation, similar to ls -lAR.
• verify compares data in the backup with actual user data, and makes sures they are identical. It
is most useful to run immediately after a backup, to check that it actually worked. It can be run
at any time, but if the user data has changed, verification fails even though the backup is OK.
• forget removes backup generations that are no longer wanted, so that they don't use disk space.
Note that after a backup generation is removed the data can't be restored anymore. You can either
specify the generations to remove by listing them on the command line, or use the --keep option to
specify a policy for what to keep (everything else will be removed).
• fsck checks the internal consistency of the backup repository. It verifies that all clients, gen‐
erations, directories, files, and all file contents still exists in the backup repository. It may
take quite a long time to run.
• force-lock removes a lock file for a client in the repository. You should only force a lock if
you are sure no-one is accessing that client's data in the repository. A dangling lock might hap‐
pen, for example, if obnam loses its network connection to the backup repository.
• client-keys lists the encryption key associated with each client.
• list-keys lists the keys that can access the repository, and which toplevel directories each key
can access. Some of the toplevel directories are shared between clients, others are specific to a
client.
• list-toplevels is like list-keys, but lists toplevels and which keys can access them.
• add-key adds an encryption key to the repository. By default, the key is added only to the shared
toplevel directories, but it can also be added to specific clients: list the names of the clients
on the command line. They key is given with the --keyid option. Whoever has access to the secret
key corresponding to the key id can access the backup repository (the shared toplevels plus speci‐
fied clients).
• remove-key removes a key from the shared toplevel directories, plus any clients specified on the
command line.
• nagios-last-backup-age is a check that exits with non-zero return if a backup age exceeds a cer‐
tain threshold. It is suitable for use as a check plugin for nagios. Thresholds can be given the
--warn-age and --critical-age options.
• diff compares two generations and lists files differing between them. Every output line will be
prefixed either by a plus sign (+) for files that were added, a minus sign (-) for files that have
been removed or an asterisk (*) for files that have changed. If only one generation ID is speci‐
fied on the command line that generation will be compared with its direct predecessor. If two IDs
have been specified, all changes between those two generations will be listed.
• mount makes the backup repository available via a read-only FUSE filesystem. This means you can
look at backed up data using normal tools, such as your GUI file manager, or command line tools
such as ls(1), diff(1), and cp(1). You can't make new backups with the mount subcommand, but you
can restore data easily.
You need to have the FUSE utilities and have permission to use FUSE for this to work. The details
will vary between operating systems; in Debian, install the package fuse and add yourself to the
fuse group (you may need to log out and back in again).
Making backups
When you run a backup, obnam uploads data into the backup repository. The data is divided into chunks,
and if a chunk already exists in the backup repository, it is not uploaded again. This allows obnam to
deal with files that have been changed or renamed since the previous backup run. It also allows several
backup clients to avoid uploading the same data. If, for example, everyone in the office has a copy of
the same sales brochures, only one copy needs to be stored in the backup repository.
Every backup run is a generation. In addition, obnam will make checkpoint generations every now and
then. These are exactly like normal generations, but are not guaranteed to be a complete snapshot of the
live data. If the backup run needs to be aborted in the middle, the next backup run can continue from
the latest checkpoint, avoiding the need to start completely over.
If one backup run drops a backup root directory, the older generations will still keep it: nothing
changes in the old generations just because there is a new one. If the root was dropped by mistake, it
can be added back and the next backup run will re-use the existing data in the backup repository, and
will only back up the file metadata (filenames, permissions, etc).
Verifying backups
What good is a backup system you cannot rely on? How can you rely on something you cannot test? The ob‐
nam verify command checks that data in the backup repository matches actual user data. It retrieves one
or more files from the repository and compares them to the user data. This is essentialy the same as do‐
ing a restore, then comparing restored files with the original files using cmp(1), but easier to use.
By default verification happens on all files. You can also specify the files to be verified by listing
them on the command line. You should specify the full paths to the files, not relative to the current
directory.
The output lists files that fail verification for some reason. If you verify everything, it is likely
that some files (e.g., parent directories of backup root) may have changed without it being a problem.
Note that you will need to specify the whole path to the files or directories to be verified, not rela‐
tive to the backup root. You still need to specify at least one of the backup roots on the command line
or via the --root option so that obnam will find the filesystem, in case it is a remote one.
URL syntax
Whenever obnam accepts a URL, it can be either a local pathname, or an sftp URL. An sftp URL has the
following form:
sftp://[user@]domain[:port]/path
where domain is a normal Internet domain name for a server, user is your username on that server, port is
an optional port number, and path is a pathname on the server side. Like bzr(1), but unlike the sftp URL
standard, the pathname is absolute, unless it starts with /~/ in which case it is relative to the user's
home directory on the server.
See the EXAMPLE section for examples of URLs.
You can use sftp URLs for the repository, or the live data (root), but note that due to limitations in
the protocol, and its implementation in the paramiko library, some things will not work very well for ac‐
cessing live data over sftp. Most importantly, the handling of of hardlinks is rather suboptimal. For
live data access, you should not end the URL with /~/ and should append a dot at the end in this special
case.
Generation specifications
When not using the latest generation, you will need to specify which one you need. This will be done
with the --generation option, which takes a generation specification as its argument. The specification
is either the word latest, meaning the latest generation (also the default), or a number. See the gener‐
ations command to see what generations are available, and what their numbers are.
Policy for keeping and removing backup generations
The forget command can follow a policy to automatically keep some and remove other backup generations.
The policy is set with the --keep=POLICY option.
POLICY is comma-separated list of rules. Each rule consists of a count and a time period. The time pe‐
riods are h, d, w, m, and y, for hour, day, week, month, and year.
A policy of 30d means to keep the latest backup for each day when a backup was made, and keep the last 30
such backups. Any backup matched by any policy rule is kept, and any backups in between will be removed,
as will any backups older than the oldest kept backup.
As an example, assume backups are taken every hour, on the hour: at 00:00, 01:00, 02:00, and so on, until
23:00. If the forget command is run at 23:15, with the above policy, it will keep the backup taken at
23:00 on each day, and remove every other backup that day. It will also remove backups older than 30
days.
If backups are made every other day, at noon, forget would keep the 30 last backups, or 60 days worth of
backups, with the above policy.
Note that obnam will only inspect timestamps in the backup repository, and does not care what the actual
current time is. This means that if you stop making new backups, the existing ones won't be removed au‐
tomatically. In essence, obnam pretends the current time is just after the latest backup when forget is
run.
The rules can be given in any order, but will be sorted to ascending order of time period before applied.
(It is an error to give two rules for the same period.) A backup generation is kept if it matches any
rule.
For example, assume the same backup frequence as above, but a policy of 30d,52w. This will keep the
newest daily backup for each day for thirty days, and the newest weekly backup for 52 weeks. Because the
hourly backups will be removed daily, before they have a chance to get saved by a weekly rule, the effect
is that the 23:00 o'clock backup for each day is saved for a month, and the 23:00 backup on Sundays is
saved for a year.
If, instead, you use a policy of 72h,30d,52w, obnam would keep the last 72 hourly backups, and the last
backup of each calendar day for 30 days, and the last backup of each calendar week for 52 weeks. If the
backup frequency was once per day, obnam would keep the backup of each calendar hour for which a backup
was made, for 72 such backups. In other words, it would effectively keep the the last 72 daily backups.
Sound confusing? Just wonder how confused the developer was when writing the code.
If no policy is given, forget will keep everything.
A typical policy might be 72h,7d,5w,12m, which means: keep the last 72 hourly backups, the last 7 daily
backups, the last 5 weekly backups and the last 12 monthly backups. If the backups are systematically
run on an hourly basis, this will mean keeping hourly backups for three days, daily backups for a week,
weekly backups for a month, and monthly backups for a year.
The way the policy works is a bit complicated. Run forget with the --pretend option to make sure you're
removing the right ones.
Using encryption
obnam can encrypt all the data it writes to the backup repository. It uses gpg(1) to do the encryption.
You need to create a key pair using gpg --gen-key (or use an existing one), and then tell obnam about it
using the --encrypt-with option.
Configuration files
obnam will look for configuration files in a number of locations. See the FILES section for a list. All
files are treated as if they were one with the contents of all files catenated.
The files are in INI format, and only the [config] section is used (any other sections are ignored).
The long names of options are used as keys for configuration variables. Any setting that can be set from
the command line can be set in a configuration file, in the [config] section.
For example, the options in the following command line:
obnam --repository=/backup --exclude='.wav$' backup
could be replaced with the following configuration file:
[config]
repository: /backup
exclude: .wav$
(You can use either foo=value or foo: value syntax in the files.)
The only unusual thing about the files is the way options that can be used many times are expressed. All
values are put in a single logical line, separated by commas (and optionally spaces as well). For exam‐
ple:
[config]
exclude = foo, bar, \.mp3$
The above has three values for the exclude option: any files that contain the words foo or bar anywhere
in the fully qualified pathname, or files with names ending with a period and mp3 (because the exclusions
are regular expressions).
A long logical line can be broken into several physical ones, by starting a new line at white space, and
indenting the continuation lines:
[config]
exclude = foo,
bar,
\.mp3$
The above example adds three exclusion patterns.
Multiple clients and locking
obnam supports sharing a repository between multiple clients. The clients can share the file contents
(chunks), so that if client A backs up a large file, and client B has the same file, then B does not need
to upload the large file to the repository a second time. For this to work without confusion, the
clients use a simple locking protocol that allows only one client at a time to modify the shared data
structures. Locks do not prevent read-only access at the same time: this allows you to restore while
someone else is backing up.
Sometimes a read-only operation happens to access a data structure at the same time as it is being modi‐
fied. This can result in a crash. It will not result in corrupt data, or incorrect restores. However,
you may need to restart the read-only operation after a crash.
Repository format conversion
The convert5to6 subcommand converts a repository of format 5 to format 6. It is somewhat dangerous! It
modifies the repository in place, so you should be careful. You should do a hardlink copy of the reposi‐
tory before converting:
cp -al repo repo.format5
You should also run this with local filesystem access to the repository, rather than sftp, to avoid
abysmal performance.
OPTIONS
--client-name=CLIENT-NAME
name of client (komainu)
--compress-with=PROGRAM
use PROGRAM to compress repository with (one of none, deflate)
--critical-age=AGE
for nagios-last-backup-age: maximum age (by default in hours) for the most recent backup before
statis is critical. Accepts one char unit specifier (s,m,h,d for seconds, minutes, hours, and
days.
--generate-manpage=TEMPLATE
SUPPRESSHELP
--generation=GENERATION
which generation to restore
-h, --help
show this help message and exit
--keep=KEEP
policy for what generations to keep when forgetting
--lock-timeout=TIMEOUT
when locking in the backup repository, wait TIMEOUT seconds for an existing lock to go away before
giving up
--no-pretend, --no-dry-run, --no-no-act
--no-quiet
--no-verbose
--output=FILE
write output to FILE, instead of standard output
--pretend, --dry-run, --no-act
do not actually change anything (works with backup, forget and restore only, and may only simulate
approximately real behavior)
--quiet
be silent
-r, --repository=REPOSITORY
name of backup repository
--root=ROOT
what to backup
--testing-fail-matching=REGEXP
development testing helper: simulate failures during backup for files that match the given regular
expressions
--to=TO
where to restore
--trace=TRACE
add to filename patters for which trace debugging logging happens
--verbose
be verbose
--verify-randomly=N
verify N files randomly from the backup (default is zero, meaning everything)
--version
show program's version number and exit
--warn-age=AGE
for nagios-last-backup-age: maximum age (by default in hours) for the most recent backup before
status is warning. Accepts one char unit specifier (s,m,h,d for seconds, minutes, hours, and days.
Backing up
--checkpoint=SIZE
make a checkpoint after a given SIZE (1073741824)
--deduplicate=MODE
find duplicate data in backed up data and store it only once; three modes are available: never de-
duplicate, verify that no hash collisions happen, or (the default) fatalistically accept the risk
of collisions
--exclude=EXCLUDE
regular expression for pathnames to exclude from backup (can be used multiple times)
--exclude-caches
exclude directories (and their subdirs) that contain a CACHEDIR.TAG file
--leave-checkpoints
leave checkpoint generations at the end of a successful backup run
--no-exclude-caches
--no-leave-checkpoints
--no-one-file-system
--no-small-files-in-btree
--one-file-system
exclude directories (and their subdirs) that are in a different filesystem
--small-files-in-btree
put contents of small files directly into the per-client B-tree, instead of separate chunk files;
do not use this as it is quite bad for performance
Configuration files and settings
--config=FILE
add FILE to config files
--dump-config
write out the entire current configuration
--dump-setting-names
SUPPRESSHELP
--help-all
show all options
--list-config-files
SUPPRESSHELP
--no-default-configs
clear list of configuration files to read
Development of Obnam itself
--crash-limit=COUNTER
artificially crash the program after COUNTER files written to the repository; this is useful for
crash testing the application, and should not be enabled for real use; set to 0 to disable (dis‐
abled by default)
--pretend-time=TIMESTAMP
pretend it is TIMESTAMP (YYYY-MM-DD HH:MM:SS); this is only useful for testing purposes
--sftp-delay=SFTP-DELAY
add an artificial delay (in milliseconds) to all SFTP transfers
Encryption
--encrypt-with=ENCRYPT-WITH
PGP key with which to encrypt data in the backup repository
--key-details
show additional user IDs for all keys
--keyid=KEYID
PGP key id to add to/remove from the backup repository
--no-key-details
--no-weak-random
--symmetric-key-bits=SYMMETRIC-KEY-BITS
size of symmetric key, in bits
--weak-random
use /dev/urandom instead of /dev/random to generate symmetric keys
Integrity checking (fsck)
--fsck-fix
should fsck try to fix problems?
--fsck-ignore-chunks
ignore chunks when checking repository integrity (assume all chunks exist and are correct)
--fsck-ignore-client=NAME
do not check repository data for cient NAME
--fsck-last-generation-only
check only the last generation for each client
--fsck-skip-dirs
do not check anything about directories and their files
--fsck-skip-files
do not check anything about files
--fsck-skip-generations
do not check any generations
--fsck-skip-per-client-b-trees
do not check per-client B-trees
--fsck-skip-shared-b-trees
do not check shared B-trees
--no-fsck-fix
--no-fsck-ignore-chunks
--no-fsck-last-generation-only
--no-fsck-skip-dirs
--no-fsck-skip-files
--no-fsck-skip-generations
--no-fsck-skip-per-client-b-trees
--no-fsck-skip-shared-b-trees
Logging
--log=FILE
write log entries to FILE (default is to not write log files at all); use "syslog" to log to sys‐
tem log, or "none" to disable logging
--log-keep=N
keep last N logs (10)
--log-level=LEVEL
log at LEVEL, one of debug, info, warning, error, critical, fatal (default: info)
--log-max=SIZE
rotate logs larger than SIZE, zero for never (default: 0)
--log-mode=MODE
set permissions of new log files to MODE (octal; default 0600)
Mounting with FUSE
--fuse-opt=FUSE
options to pass directly to Fuse
--viewmode=MODE
"single" directly mount specified generation, "multiple" mount all generations as separate direc‐
tories
Peformance
--dump-memory-profile=METHOD
make memory profiling dumps using METHOD, which is one of: none, simple, meliae, or heapy (de‐
fault: simple)
--memory-dump-interval=SECONDS
make memory profiling dumps at least SECONDS apart
Performance tweaking
--chunk-size=SIZE
size of chunks of file data backed up (default: 1048576)
--chunkids-per-group=NUM
encode NUM chunk ids per group (1024)
--idpath-bits=IDPATH-BITS
chunk id level size
--idpath-depth=IDPATH-DEPTH
depth of chunk id mapping
--idpath-skip=IDPATH-SKIP
chunk id mapping lowest bits skip
--lru-size=SIZE
size of LRU cache for B-tree nodes (default: 256)
--node-size=SIZE
size of B-tree nodes on disk; only affects new B-trees so you may need to delete a client or
repository to change this for existing repositories (default: 262144)
--upload-queue-size=SIZE
length of upload queue for B-tree nodes (default: 128)
SSH/SFTP
--no-pure-paramiko
--no-strict-ssh-host-keys
--pure-paramiko
do not use openssh even if available, use paramiko only instead
--ssh-command=EXECUTABLE
alternative executable to be used instead of "ssh" (full path is allowed, no arguments may be
added)
--ssh-host-keys-check=VALUE
If "yes", require that the ssh host key must be known and correct to be accepted. If "no", do not
require that. If "ask", the user is interactively asked to accept new hosts. The default ("ssh-
config") is to rely on the settings of the underlying SSH client
--ssh-key=FILENAME
use FILENAME as the ssh RSA private key for sftp access (default is using keys known to ssh-agent)
--ssh-known-hosts=FILENAME
filename of the user's known hosts file (default: /home/buildd/.ssh/known_hosts)
--strict-ssh-host-keys
DEPRECATED, use --ssh-host-keys-check instead
Option values
The SIZE value in options mentioned above specifies a size in bytes, with optional suffixes to indicate
kilobytes (k), kibibytes (Ki), megabytes (M), mebibyts (Mi), gigabytes (G), gibibytes (Gi), terabytes
(T), tibibytes (Ti). The suffixes are case-insensitive.
EXIT STATUS
obnam will exit with zero if everything went well, and non-zero otherwise.
ENVIRONMENT
obnam will pass on the environment it gets from its parent, without modification. It does not obey any
unusual environment variables, but it does obey the usual ones when running external programs, creating
temporary files, etc.
FILES
/etc/obnam.conf
/etc/obnam/*.conf
~/.obnam.conf
~/.config/obnam/*.conf
Configuration files for obnam. It is not an error for any or all of the files to not exist.
EXAMPLE
To back up your home directory to a server:
obnam backup --repository sftp://your.server/~/backups $HOME
To restore your latest backup from the server:
obnam restore --repository sftp://your.server/~/backups \
--to /var/tmp/my.home.dir
To restore just one file or directory:
obnam restore --repository sftp://your.server/~/backups \
--to /var/tmp/my.home.dir $HOME/myfile.txt
Alternatively, mount the backup repository using the FUSE filesystem (note that the --to option is neces‐
sary and that the --viewmode option is usually a good idea):
mkdir my-repo
obnam restore --repository sftp://your.server/~/backups \
--to my-repo --viewmode multiple
cp my-repo/latest/$HOME/myfile.txt
fusermount -u my-repo
To check that the backup worked:
obnam verify --repository sftp://your.server/~/backups \
/path/to/file
To remove old backups, keeping the newest backup for each day for
ten years:
obnam forget --repository sftp://your.server/~/backups \
--keep 3650d
To verify that the backup repository is OK:
obnam fsck --repository sftp://your.server/~/backups
To view the backed up files in the backup repository using FUSE:
obnam mount --to my-fuse --viewmode multiple
ls -lh my-fuse
fusermount -u my-fuse
SEE ALSO
cliapp(5)
OBNAM(1)