bionic (1) pegasus-transfer.1.gz

NAME

       pegasus-transfer - Handles data transfers in Pegasus workflows.

SYNOPSIS

       pegasus-transfer [-h]
                          [--file inputfile]
                          [--threads number_threads]
                          [--max-attempts attempts]
                          [--debug]

DESCRIPTION

       pegasus-transfer takes a list of url pairs, either on stdin or with an input file, determines the correct
       tool to use for the transfer and executes the transfer. Some of the protocols pegasus-transfer can handle
       are GridFTP, SRM, Amazon S3, HTTP, and local cp/symlinking. Failed transfers are retried.

       Note that pegasus-transfer is a tool mostly used internally in Pegasus workflows, but the tool can be
       used stand alone as well.

OPTIONS

       -h, --help
           Prints a usage summary with all the available command-line options.

       -f inputfile, --file inputfile
           File with input pairs. If not given, stdin will be used.

       -m, --max-attempts attempts
           Maximum number of attempts for retrying failed transfers.

       -t, --threads number_threads
           The number of threads to use. This controls the parallelism of transfers.

       -d, --debug
           Enables debugging output.

EXAMPLE

           $ pegasus-transfer
           # src 1 local_site
           file:///etc/hosts
           # dst 1 local_site
           file:///tmp/foo
           CTRL+D

CREDENTIAL HANDLING

       Credentials used for transfers can be specified with a combination of comments in the input file format
       and environment variables. For example, give the following input file:

           # src 1 isi
           gsiftp://workflow.isi.edu/data/file.dat
           # dst 1 tacc_stampede
           gsiftp://gridftp.stampede.tacc.utexas.edu/scratch/file.dat

       pegasus-transfer will expect either one environment variable specifying one credential to be used on both
       end of the connection (X509_USER_PROXY), or two separate environment variables specifying two different
       credentials to be used on the two ends of the connection. The the latter case, the environment variables
       are derived from the site names provided in the input file commments. In the example above, the
       environment variables would be named X509_USER_PROXY_isi and X509_USER_PROXY_tacc_stampede

THREADING

       In order to speed up data transfers, pegasus-transfer will start a set of transfers in parallel using
       threads. Threads are turned off when retrying failed transfers.

AUTHOR

       Pegasus Team http://pegasus.isi.edu