Ubuntu Manpage: datalad addurls - create and update a dataset from a list of URLs.

NAME

       datalad addurls - create and update a dataset from a list of URLs.

SYNOPSIS

       datalad  addurls  [-h]  [-d  DATASET]  [-t  TYPE]  [-x  REGEXP] [-m FORMAT] [--key FORMAT]
              [--message  MESSAGE]  [-n]  [--fast]  [--ifexists   {overwrite|skip}]   [--missing-
              value VALUE] [--nosave] [--version-urls] [-c PROC] [-J NJOBS] [--drop-after] [--on-
              collision  {error|error-if-different|take-first|take-last}]  [--version]   URL-FILE
              URL-FORMAT FILENAME-FORMAT

DESCRIPTION

Format specification
Several arguments take format strings. These are similar to normal Python format strings
where the names from `URL-FILE` (column names for a comma- or tab-separated file or
properties for JSON) are available as placeholders. If `URL-FILE` is a CSV or TSV file, a
positional index can also be used (i.e., "{0}" for the first column). Note that a
placeholder cannot contain a ':' or '!'.

In addition, the `FILENAME-FORMAT` arguments has a few special placeholders.

- _repindex

The constructed file names must be unique across all fields rows. To
avoid collisions, the special placeholder "_repindex" can be added to
the formatter. Its value will start at 0 and increment every time a
file name repeats.

- _url_hostname, _urlN, _url_basename*

Various parts of the formatted URL are available. Take
"http://datalad.org/asciicast/seamless_nested_repos.sh" as an example.

"datalad.org" is stored as "_url_hostname". Components of the URL's
path can be referenced as "_urlN". "_url0" and "_url1" would map to
"asciicast" and "seamless_nested_repos.sh", respectively. The final
part of the path is also available as "_url_basename".

This name is broken down further. "_url_basename_root" and
"_url_basename_ext" provide access to the root name and extension.
These values are similar to the result of os.path.splitext, but, in the
case of multiple periods, the extension is identified using the same
length heuristic that git-annex uses. As a result, the extension of
"file.tar.gz" would be ".tar.gz", not ".gz". In addition, the fields
"_url_basename_root_py" and "_url_basename_ext_py" provide access to
the result of os.path.splitext.

- _url_filename*

These are similar to _url_basename* fields, but they are obtained with
a server request. This is useful if the file name is set in the
Content-Disposition header.

Examples
Consider a file "avatars.csv" that contains::

who,ext,link
neurodebian,png,https://avatars3.githubusercontent.com/u/260793
datalad,png,https://avatars1.githubusercontent.com/u/8927200

To download each link into a file name composed of the 'who' and 'ext' fields, we could
run::

$ datalad addurls -d avatar_ds avatars.csv '{link}' '{who}.{ext}'

The `-d avatar_ds` is used to create a new dataset in "$PWD/avatar_ds".

If we were already in a dataset and wanted to create a new subdataset in an "avatars"
subdirectory, we could use "//" in the `FILENAME-FORMAT` argument::

$ datalad addurls avatars.csv '{link}' 'avatars//{who}.{ext}'

If the information is represented as JSON lines instead of comma separated values or a
JSON array, you can use a utility like jq to transform the JSON lines into an array that
addurls accepts::

$ ... | jq --slurp . | datalad addurls - '{link}' '{who}.{ext}'

NOTE

For users familiar with 'git annex addurl': A large part of this
plugin's functionality can be viewed as transforming data from
`URL-FILE` into a "url filename" format that fed to 'git annex addurl
--batch --with-files'.

OPTIONS

URL-FILE
A file that contains URLs or information that can be used to construct URLs.
Depending on the value of --input-type, this should be a comma- or tab-separated
file (with a header as the first row) or a JSON file (structured as a list of
objects with string values). If '-', read from standard input, taking the content
as JSON when --input-type is at its default value of 'ext'.

URL-FORMAT
A format string that specifies the URL for each entry. See the 'Format
Specification' section above.

FILENAME-FORMAT
Like `URL-FORMAT`, but this format string specifies the file to which the URL's
content will be downloaded. The name should be a relative path and will be taken as
relative to the top-level dataset, regardless of whether it is specified via
--dataset or inferred. The file name may contain directories. The separator "//"
can be used to indicate that the left-side directory should be created as a new
subdataset. See the 'Format Specification' section above.

-h, --help, --help-np
show this help message. --help-np forcefully disables the use of a pager for
displaying the help message

-d DATASET, --dataset DATASET
Add the URLs to this dataset (or possibly subdatasets of this dataset). An empty or
non-existent directory is passed to create a new dataset. New subdatasets can be
specified with `FILENAME-FORMAT`. Constraints: Value must be a Dataset or a valid
identifier of a Dataset (e.g. a path) or value must be NONE

-t TYPE, --input-type TYPE
Whether `URL-FILE` should be considered a CSV file, TSV file, or JSON file. The
default value, "ext", means to consider `URL-FILE` as a JSON file if it ends with
".json" or a TSV file if it ends with ".tsv". Otherwise, treat it as a CSV file.
Constraints: value must be one of ('ext', 'csv', 'tsv', 'json') [Default: 'ext']

-x REGEXP, --exclude-autometa REGEXP
By default, metadata field=value pairs are constructed with each column in `URL-
FILE`, excluding any single column that is specified via `URL-FORMAT`. This
argument can be used to exclude columns that match a regular expression. If set to
'*' or an empty string, automatic metadata extraction is disabled completely. This
argument does not affect metadata set explicitly with --meta.

-m FORMAT, --meta FORMAT
A format string that specifies metadata. It should be structured as
"<field>=<value>". As an example, "location={3}" would mean that the value for the
"location" metadata field should be set the value of the fourth column. This option
can be given multiple times.

--key FORMAT
A format string that specifies an annex key for the file content. In this case, the
file is not downloaded; instead the key is used to create the file without content.
The value should be structured as "[et:]<input backend>[-s<bytes>]--<hash>". The
optional "et:" prefix, which requires git-annex 8.20201116 or later, signals to
toggle extension state of the input backend (i.e., MD5 vs MD5E). As an example,
"et:MD5-s{size}--{md5sum}" would use the 'md5sum' and 'size' columns to construct
the key, migrating the key from MD5 to MD5E, with an extension based on the file
name. Note: If the *input* backend itself is an annex extension backend (i.e., a
backend with a trailing "E"), the key's extension will not be updated to match the
extension of the corresponding file name. Thus, unless the input keys and file
names are generated from git-annex, it is recommended to avoid using extension
backends as input. If an extension is desired, use the plain variant as input and
prepend "et:" so that git-annex will migrate from the plain backend to the
extension variant.

--message MESSAGE
Use this message when committing the URL additions. Constraints: value must be NONE
or value must be a string

-n, --dry-run
Report which URLs would be downloaded to which files and then exit.

--fast If True, add the URLs, but don't download their content. WARNING: ONLY USE THIS
OPTION IF YOU UNDERSTAND THE CONSEQUENCES. If the content of the URLs is not
downloaded, then datalad will refuse to retrieve the contents with `datalad get
<file>` by default because the content of the URLs is not verified. Add
`annex.security.allow-unverified-downloads = ACKTHPPT` to your git config to bypass
the safety check. Underneath, this passes the `--fast` flag to `git annex addurl`.

--ifexists {overwrite|skip}
What to do if a constructed file name already exists. The default behavior is to
proceed with the `git annex addurl`, which will fail if the file size has changed.
If set to 'overwrite', remove the old file before adding the new one. If set to
'skip', do not add the new file. Constraints: value must be one of ('overwrite',
'skip')

--missing-value VALUE
When an empty string is encountered, use this value instead. Constraints: value
must be NONE or value must be a string

--nosave
by default all modifications to a dataset are immediately saved. Giving this option
will disable this behavior.

--version-urls
Try to add a version ID to the URL. This currently only has an effect on HTTP URLs
for AWS S3 buckets. s3:// URL versioning is not yet supported, but any URL that
already contains a "versionId=" parameter will be used as is.

-c PROC, --cfg-proc PROC
Pass this --cfg_proc value when calling CREATE to make datasets.

-J NJOBS, --jobs NJOBS
how many parallel jobs (where possible) to use. "auto" corresponds to the number
defined by 'datalad.runtime.max-annex-jobs' configuration item. Constraints: value
must be convertible to type 'int' or value must be NONE or value must be one of
('auto',)

--drop-after
drop files after adding to annex.

--on-collision {error|error-if-different|take-first|take-last}
What to do when more than one row produces the same file name. By default an error
is triggered. "error-if-different" suppresses that error if rows for a given file
name collision have the same URL and metadata. "take-first" or "take-last" indicate
to instead take the first row or last row from each set of colliding rows.
Constraints: value must be one of ('error', 'error-if-different', 'take-first',
'take-last') [Default: 'error']

--version
show the module and its version which provides the command

AUTHORS

        datalad is developed by The DataLad Team and Contributors <team@datalad.org>.