NAME

       datalad addurls - create and update a dataset from a list of URLs.

SYNOPSIS

       datalad  addurls  [-h]  [-d DATASET] [-t TYPE] [-x REGEXP] [-m FORMAT] [--message MESSAGE]
              [-n] [--fast]  [--ifexists  {overwrite|skip}]  [--missing-value  VALUE]  [--nosave]
              [--version-urls] [-c PROC] URL-FILE URL-FORMAT FILENAME-FORMAT

DESCRIPTION

   Format specification
       Several  arguments take format strings.  These are similar to normal Python format strings
       where the names from URL-FILE (column  names  for  a  CSV  or  properties  for  JSON)  are
       available as placeholders.  If URL-FILE is a CSV file, a positional index can also be used
       (i.e., "{0}" for the first column).  Note that a placeholder cannot contain a ':' or '!'.

       In addition, the FILENAME-FORMAT arguments has a few special placeholders.

       - _repindex

         The constructed file names must be unique across all fields rows.  To
         avoid collisions, the special placeholder "_repindex" can be added to
         the formatter.  Its value will start at 0 and increment every time a
         file name repeats.

       - _url_hostname, _urlN, _url_basename*

         Various parts of the formatted URL are available.  Take
         "http://datalad.org/asciicast/seamless_nested_repos.sh" as an example.

         "datalad.org" is stored as "_url_hostname".  Components of the URL's
         path can be referenced as "_urlN".  "_url0" and "_url1" would map to
         "asciicast" and "seamless_nested_repos.sh", respectively.  The final
         part of the path is also available as "_url_basename".

         This name is broken down further.  "_url_basename_root" and
         "_url_basename_ext" provide access to the root name and extension.
         These values are similar to the result of os.path.splitext, but, in the
         case of multiple periods, the extension is identified using the same
         length heuristic that git-annex uses.  As a result, the extension of
         "file.tar.gz" would be ".tar.gz", not ".gz".  In addition, the fields
         "_url_basename_root_py" and "_url_basename_ext_py" provide access to
         the result of os.path.splitext.

       - _url_filename*

         These are similar to _url_basename* fields, but they are obtained with
         a server request.  This is useful if the file name is set in the
         Content-Disposition header.

   Examples
       Consider a file "avatars.csv" that contains::

         who,ext,link
         neurodebian,png,https://avatars3.githubusercontent.com/u/260793
         datalad,png,https://avatars1.githubusercontent.com/u/8927200

       To download each link into a file name composed of the 'who' and 'ext'  fields,  we  could
       run::

       $ datalad addurls -d avatar_ds --fast avatars.csv '{link}' '{who}.{ext}'

       The `-d avatar_ds` is used to create a new dataset in "$PWD/avatar_ds".

       If  we  were  already  in  a dataset and wanted to create a new subdataset in an "avatars"
       subdirectory, we could use "//" in the FILENAME-FORMAT argument::

       $ datalad addurls --fast avatars.csv '{link}' 'avatars//{who}.{ext}'

       NOTE

        For users familiar with 'git annex addurl': A large part of this
        plugin's functionality can be viewed as transforming data from
        URL-FILE into a "url filename" format that fed to 'git annex addurl
        --batch --with-files'.

OPTIONS

       URL-FILE
              A file that contains URLs or information  that  can  be  used  to  construct  URLs.
              Depending on the value of --input-type, this should be a CSV file (with a header as
              the first row) or a JSON file (structured as a list of objects with string values).

       URL-FORMAT
              A  format  string  that  specifies  the  URL  for  each  entry.  See  the   'Format
              Specification' section above.

       FILENAME-FORMAT
              Like  URL-FORMAT,  but  this  format  string  specifies the file to which the URL's
              content will be downloaded. The name should be a relative path and will be taken as
              relative  to  the  top-level  dataset,  regardless  of  whether it is specified via
              --dataset) or inferred. The file name may contain directories. The  separator  "//"
              can  be  used  to  indicate that the left-side directory should be created as a new
              subdataset. See the 'Format Specification' section above.

       -h, -\-help, -\-help-np
              show this help message. --help-np forcefully  disables  the  use  of  a  pager  for
              displaying the help message

       -d DATASET, -\-dataset DATASET
              Add the URLs to this dataset (or possibly subdatasets of this dataset). An empty or
              non-existent directory is passed to create a new dataset. New  subdatasets  can  be
              specified  with  FILENAME-FORMAT.  Constraints:  Value must be a Dataset or a valid
              identifier of a Dataset (e.g. a path)

       -t TYPE, -\-input-type TYPE
              Whether URL-FILE should be considered a CSV file or a JSON file. The default value,
              "ext",  means  to  consider  URL-FILE  as  a  JSON  file  if  it ends with ".json".
              Otherwise, treat it as a CSV file. Constraints: value must be one of ('ext', 'csv',
              'json') [Default: 'ext']

       -x REGEXP, -\-exclude-autometa REGEXP
              By  default,  metadata  field=value  pairs are constructed with each column in URL-
              FILE, excluding any single column that is specified via URL-FORMAT.  This  argument
              can be used to exclude columns that match a regular expression. If set to '*' or an
              empty string, automatic metadata extraction is disabled completely.  This  argument
              does not affect metadata set explicitly with --meta.

       -m FORMAT, -\-meta FORMAT
              A   format   string   that   specifies   metadata.   It  should  be  structured  as
              "<field>=<value>". As an example, "location={3}" would mean that the value for  the
              "location" metadata field should be set the value of the fourth column. This option
              can be given multiple times.

       -\-message MESSAGE
              Use this message when committing the URL  additions.  Constraints:  value  must  be
              NONE, or value must be a string

       -n, -\-dry-run
              Report which URLs would be downloaded to which files and then exit.

       -\-fast
              If  True,  add  the URLs, but don't download their content. Underneath, this passes
              the --fast flag to `git annex addurl`.

       -\-ifexists {overwrite|skip}
              What to do if a constructed file name already exists. The default  behavior  is  to
              proceed  with the `git annex addurl`, which will fail if the file size has changed.
              If set to 'overwrite', remove the old file before adding the new  one.  If  set  to
              'skip',  do  not  add  the  new  file.  Constraints:  value  must  be one of (None,
              'overwrite', 'skip')

       -\-missing-value VALUE
              When an empty string is encountered, use this  value  instead.  Constraints:  value
              must be NONE, or value must be a string

       -\-nosave
              by default all modifications to a dataset are immediately saved. Giving this option
              will disable this behavior.

       -\-version-urls
              Try to add a version ID to the URL. This currently only has an effect on HTTP  URLs
              for  AWS  S3  buckets.  s3:// URL versioning is not yet supported, but any URL that
              already contains a "versionId=" parameter will be used as is.

       -c PROC, -\-cfg-proc PROC
              Pass this --cfg_proc value when calling CREATE to make datasets.

AUTHORS

        datalad is developed by The DataLad Team and Contributors <team@datalad.org>.