NAME

     fpsync — Synchronize directories in parallel using fpart + rsync or cpio

SYNOPSIS

     fpsync [-h] [-v] [-m tool] [-n jobs] [-f files] [-s size] [-E] [-w wrks] [-d shdir]
            [-t tmpdir] [-r jobname] [-o toolopts] [-O fpartopts] [-S] src_dir/ dst_url/

DESCRIPTION

     The fpsync tool synchronizes directories in parallel using fpart(1) and rsync(1) or cpio(1).
     It computes subsets of src_dir/ and spawns jobs to synchronize them to dst_url/.

     Synchronization jobs can be executed either locally or remotely (using SSH workers, see
     option -w) and are executed on-the-fly while filesystem crawling goes on.  This makes fpsync
     a good tool for migrating large filesystems.

OPTIONS

     -h      Print help

     -v      Verbose mode.  Can be be specified several times to increase verbosity level.

     -m tool
             External copy tool used to synchronize files.  Currently supported tools are: rsync
             and cpio.  Default: rsync.  When using cpio and more than one worker, directory
             timestamps may not be replicated.  A second pass will fix them.  See the README file
             for more details.

     -n jobs
             Start jobs concurrent sync jobs (either locally or remotely, see below).  Default: 2

     -f files
             Transfer at most files files or directories per sync job.  0 means unlimited but you
             must at least specify one file or size limit.  Default: 2000

     -s size
             Transfer at most size bytes per sync job.  0 means unlimited but you must at least
             specify one file or size limit.  You can use a human-friendly unit suffix here (k,
             m, g, t, p).
             Default: 4294967296 (4 GB)

     -E      Work on a per-directory basis (rsync tool only).  In that mode, fpsync works with
             lists of directories instead of files.  That mode may generate coarse-grained lists
             but enables rsync(1) 's --delete option by default ( WARNING!!!  ), making it a good
             candidate for a final (cleaning) pass after several incremental passes using
             standard (file) mode.

     -w wrks
             Use remote SSH wrks to synchronize files.  Synchronization jobs are executed locally
             when this option is not set.  wrks is a space-separated list of login@machine
             connection strings and can be specified several times.  You must be allowed to
             connect to those machines using a SSH key to avoid user interaction.

     -d shdir
             Set fpsync shared directory to shdir.  This option is mandatory when using SSH
             workers and set by default to tmpdir when running locally.  The specified directory
             must be an absolute path ; it will be used to handle communications with SSH hosts
             (sharing partitions and log files) and, as a consequence, must be made available to
             all participating hosts (e.g. through a r/w NFS mount), including the master one
             running fpsync.

     -t tmpdir
             Set fpsync temporary directory to tmpdir.  This directory remains local and does not
             need to be shared amongst SSH workers when using the -w option.  Default:
             /tmp/fpsync

     -r jobname
             Resume job jobname and restart synchronizing remaining partitions from a previous
             run.  jobname can be obtained using verbose mode (see option -v).  Note that
             filesystem crawling is skipped when resuming a previous run.  As a consequence,
             options -f, -s, -o, -O, -S, src_dir/, and dst_url/ are ignored.

     -o toolopts
             Override default rsync(1) or cpio(1) options with toolopts.  Use this option with
             care as certain options are incompatible with a parallel usage (e.g. rsync's
             --delete).  Default for rsync: -lptgoD -v --numeric-ids.  Empty for cpio.

     -O fpartopts
             Override default fpart(1) options with fpartopts.
             Default: -x .zfs -x .snapshot* -x .ckpt

     -S      Sudo mode.  Use sudo(8) for filesystem crawling and synchronizations.

     src_dir/
             Source directory.  It must be absolute and available on all participating hosts
             (including the master one, running fpsync).

     dst_url/
             Destination directory or URL (rsync tool only).  If a remote URL is provided, it
             must be supported by rsync(1).  All participating workers must be able to reach that
             target.

RUNNING FPSYNC

     Each fpsync run generates a unique jobname, which is displayed in verbose mode (see option
     -v) and within log files.  You can use that jobname to resume a previous run (see option
     -r).  fpsync will then restart synchronizing data from the parts that were being
     synchronized at the time it stopped.

     This unique feature gives the administrator the ability to stop fpsync and restart it later,
     without having to restart the whole filesystem crawling and synchronization process.  Note
     that resuming is only possible when filesystem crawling step has finished.

     During synchronization, you can press CTRL-C to interrupt the process.  The first CTRL-C
     prevents new synchronizations from being submitted and the process will wait for current
     synchronizations to be finished before exiting.  If you press CTRL-C again, current
     synchronizations will be killed and fpsync will exit immediately.  When using option -E to
     enable directory mode and rsync's --delete option, keep in mind that killing rsync processes
     may lead to a situation where certain files have been updated and others not deleted yet
     (because the deletion process is postponed using rsync's --delete-after option).

     On certain systems, CTRL-T can be pressed to get the status of current and remaining parts
     to be synchronized.  This can also be achieved by sending a SIGINFO to the fpsync process.

     Whether you use verbose mode or not, everything is logged within shdir/log/.

EXAMPLES

     Here are some examples:

     fpsync -n 4 /usr/src/ /var/src/

             Synchronizes /usr/src/ to /var/src/ using 4 local jobs.

     fpsync -n 2 -w login@machine1 -w login@machine2 -d /mnt/fpsync /mnt/src/ /mnt/dst/

             Synchronizes /mnt/src/ to /mnt/dst/ using 2 concurrent jobs executed remotely on 2
             SSH workers (machine1 and machine2).  The shared directory is set to /mnt/fpsync and
             mounted on the machine running fpsync, as well as on machine1 and machine2.  The
             source directory (/mnt/src/) is also available on those 3 machines, while the
             destination directory (/mnt/dst/) is mounted on SSH workers only (machine1 and
             machine2).

LIMITATIONS

     Parallelizing rsync(1) or cpio(1) makes several options not usable, such as rsync's
     --delete.  If your source directory is live while fpsync is running, you will have to delete
     extra files from destination directory.  This is traditionally done by using a final
     -offline- rsync(1) pass that will use this option, but you can also use fpsync and option -E
     to perform the same task using several workers.

     fpsync enqueues synchronization jobs on disk, within the tmpdir/queue directory.  Be careful
     to host this queue on a filesystem that can handle fine-grained mtime timestamps (i.e. with
     a sub-second precision) if you want the queue to be processed in order when fpart(1)
     generates several jobs per second.  On FreeBSD, VFS(9) timestamps' precision can be tuned
     using the 'vfs.timestamp_precision' sysctl.  See vfs_timestamp(9).

     Contrary to rsync(1), fpsync enforces the final '/' on the source directory.  It means that
     directory contents are synchronized, not the source directory itself (i.e. you will not get
     a subdirectory of the name of the source directory in the target directory after
     synchronization).

SEE ALSO

     cpio(1), fpart(1), rsync(1), sudo(8)

AUTHOR, AVAILABILITY

     Fpsync has been written by Ganaël LAPLANCHE and is available under the BSD license on
     http://contribs.martymac.org

BUGS

     No bug known (yet).