Provided by: gridsite-clients_3.0.0~20180202git2fdbc6f-3_amd64 
NAME
htcp, htmv, htrm, htls, htll, htmkdir, htfind, htping - file transfers and queries via
HTTP/HTTPS/SiteCast
SYNOPSIS
htcp, htmv [options] Source-URL[s] Destination-URL
htrm, htls, htll, htmkir, htfind [options] Target-URL[s]
htping [options]
DESCRIPTION
htcp is a client to fetch files or directory listings from remote servers using HTTP or
HTTPS, or to put or delete files or directories onto remote servers using HTTPS. htcp is
similar to scp(1), but uses HTTP/HTTPS rather than ssh as its transfer protocol. htcp can
also use the HTCP protocol to query HTTP(S) fileservers via SiteCast.
When talking to a fileserver with HTTPS, htcp can run "anonymously", with a standard X.509
user certificate and key, or with a GSI Proxy. This makes htcp very useful in Grid
environments where many users have certificates and where jobs and users have access to
GSI proxies.
URLs
htcp supports the file:, http: and https: URL schemes as sources and destinations. If no
scheme is given, the URL scheme is assumed to be file: and relative to the current
directory if not an absolute path.
If multiple sources are given during a copy, they will be used in turn and the destination
must be a directory (directories are indicated by a trailing /) However, source and
destination cannot both refer to remote servers.
OPTIONS
-v/--verbose
Turn on debugging information. Used once, this option will enable htcp's messages
to stderr. Used twice, will also enable the underlying libcurl messages.
--delete
Instead of copying files, delete all the URLs given on the command line. Calling
the program as htrm has the same effect.
--list Instead of copying files, output lists of files located in the URL-directories
given on the command line. Calling the program as htls has the same effect.
--long-list
Instead of copying files, output long listings of files located in the URL-
directories given on the command line. If available, the size in bytes and
modification time of each file is given. Calling the program as htll has the same
effect.
--mkdir
Instead of copying files, attempt to create a directory on a remote server with
HTTP PUT. The server must support the convention that PUT to a URL with a trailing
slash means create a directory. No file body is sent. Calling the program as
htmkdir has the same effect.
--move Move/rename files on a single remote server, given the two, absolute URLs of the
remote file names. Server must support HTTP/WebDAV MOVE. Calling the program as
htmv has the same effect.
--ping Query specified multicast groups with the HTCP NOP ("No Operation") code. SiteCast
enabled servers will respond immediately with a NOP reply, and all of the responses
will be listed, with the round trip time in milliseconds. Any waiting times
specified in the --groups option will be ignored. Calling the program as htping has
the same effect. (--groups must be used for this option to work.)
--find Query specified multicast groups with the HTCP TST code. SiteCast enabled servers
will respond with TST replies if they have the files corresponding to the given
SiteCast target URL(s). All of the transfer URLs returned will be listed. Waiting
times specified in the --groups option will be used to space out the multicast
queries, but the program listens for responses continuously. Calling the program as
htfind has the same effect. (--groups must be used for this option to work.)
--groups <IP Groups>
IP multicast groups to use for SiteCast queries. IP Groups is a comma separated
list of groups, in the format: nnn.nnn.nnn.nnn:port[:ttl[:seconds]] The IP number
and port must be specified. The IP time-to-live, ttl, controls how many networks
the multicast packets may pass through - the default, 1, limits packets to the
local network. Multiple groups may be specified, separated by commas. If multiple
groups are specified, then seconds is the time to wait before making the next
multicast - 1 second is the default.
--timeout <seconds>
A request timeout used for multicast ping.
--anon Do not attempt to use X.509 user certificates or GSI proxies to authenticate to the
remote HTTPS server. This means you are "anonymous", but the server's identity may
still be verified and the connection is still encrypted.
--cert <X.509 cert path> and --key <X.509 key path>
Path to the PEM-encoded X.509 or GSI Proxy user certificate and key to use for
HTTPS connections, instead of "anonymous mode." If only one of --key or --cert is
given, then that will be tried for both. If neither is given, then the following
order of precedence is used: the file name held by the variable X509_USER_PROXY;
the file /tmp/x509up_uID (with Unix UID equal to ID); the file names held by
X509_USER_CERT / X509_USER_KEY; the files ~/.globus/usercert.pem and
~/.globus/userkey.pem (where ~/ is the home directory of the user.)
--capath <X.509 CA root certs directory or file>
Path to the PEM-encoded CA root certificates to use when verifying remote servers'
host certificates in HTTPS connections. Ideally this should be a directory of
hash.0 files as described in the OpenSSL verify(1) man page, but a file may be used
instead. If --capath is not given, the value of the environment variable
X509_CERT_DIR will be tried. If this is not valid, then /etc/grid-
security/certificates will be used.
--no-verify
Do not use CA root certificates to verify remote servers' host certificates. This
is useful for testing sites before their certificate is set up properly, but leaves
you vulnerable to "man in the middle" attacks by hostile servers masquerading as
your target.
--grid-http
Try to use GridHTTP redirection for HTTPS URLs. Compatible servers will perform
authentication and authorization on the HTTPS connection and then redirect to HTTP
for the GET or PUT file transfer. htcp makes the HTTP request using the
GRID_AUTH_PASSCODE single-use passcode obtained via HTTPS. The --grid-http option
will be ignored for directory operations or HTTP URLs. If a redirected transfer
isn't possible, a normal HTTPS data transfer will be attempted.
--sitecast
Try to use SiteCast to locate remote files which are to be copied (currently only
for the fetching of remote files.) If no location is found via SiteCast, then a
direct request for the given URL is tried. (--groups must be used for this option
to work.)
--domain <SiteCast domain>
Try to use SiteCast to locate remote files which are to be copied (currently only
for the fetching of remote files) if the domain component of the URL matches the
SiteCast domain given. If no location is found via SiteCast, then a direct request
for the given URL is tried. (--groups must be used for this option to work.)
FILES
/tmp/x509up_uID
Default GSI Proxy file for Unix UID equal to ID.
/etc/grid-security/certificates
Default location for trusted Certification Authority root certificates to use when
checking server certificates.
/tmp/.ca-roots-XXXXXX
Prior to 7.9.8, the underlying curl library did not support the CA root
certificates directory. If built with an old version of libcurl, htcp will
concatenate the certificates in the CA roots directory into a unique temporary file
and use that.
ENVIRONMENT
X509_CERT_DIR
Holds directory to search for Certification Authority root certificates when
verifying server certificates. (Tried if --capath is not given on the command
line.)
X509_USER_PROXY
Holds file name of a GSI Proxy to use as user certificate. (Tried if --cert or
--key are not given on the command line.)
X509_USER_CERT and X509_USER_KEY
Holds file name of X.509 user certificate and key. (Tried if X509_USER_PROXY is not
valid.)
EXIT CODES
0 is returned on complete success. Curl error codes are returned when reported by the
underlying curl library, and CURLE_HTTP_RETURNED_ERROR (22) is returned when the HTTP(S)
server returns a code outside the range 200-299. The manpage libcurl-errors(3) lists all
the curl error codes.
TO DO
Recursive copying. Server-side wildcards. Parallel streams. Better error recovery.
AUTHOR
Andrew McNab <Andrew.McNab@manchester.ac.uk>
htcp is part of GridSite: http://www.gridsite.org/
SEE ALSO
scp(1), curl(1), wget(1), verify(1), libcurl-errors(3)