Provided by: datalad_0.15.5-1_all bug

NAME

       datalad search - search dataset metadata

SYNOPSIS

       datalad  search  [-h]  [-d  DATASET]  [--reindex]  [--max-nresults  MAX_NRESULTS]  [--mode
              {egrep,textblob,autofield}]   [--full-record]    [--show-keys    {name,short,full}]
              [--show-query] [--version] [QUERY ...]

DESCRIPTION

       DataLad can search metadata extracted from a dataset and/or aggregated into a superdataset
       (see the AGGREGATE-METADATA command). This makes it  possible  to  discover  datasets,  or
       individual files in a dataset even when they are not available locally.

       Ultimately  DataLad  metadata are a graph of linked data structures. However, this command
       does not (yet) support queries that can exploit all information stored in the metadata. At
       the  moment the following search modes are implemented that represent different trade-offs
       between the expressiveness of a query and the computational and storage resources required
       to execute a query.

       - egrep (default)

       - egrepcs [case-sensitive egrep]

       - textblob

       - autofield

       An  alternative  default  mode  can  be  configured  by  tuning the configuration variable
       'datalad.search.default-mode'::

       [datalad "search"]
         default-mode = egrepcs

       Each search mode has its own default configuration for what kind of  documents  to  query.
       The respective default can be changed via configuration variables::

       [datalad "search"]
         index-<mode_name>-documenttype = (all|datasets|files)

   Mode: egrep/egrepcs
       These  search  modes  are  largely  ignorant of the metadata structure, and simply perform
       matching of a search pattern against a flat string-representation  of  metadata.  This  is
       advantageous  when  the  query  is  simple  and  the  metadata structure is irrelevant, or
       precisely known.  Moreover, it does not require a  search  index,  hence  results  can  be
       reported  without  an  initial  latency  for  building  a search index when the underlying
       metadata has changed (e.g. due to a dataset update). By default, these search  modes  only
       consider  datasets  and do not investigate records for individual files for speed reasons.
       Search results are reported in the order in which they were discovered.

       Queries     can     make     use     of     Python     regular      expression      syntax
       (https://docs.python.org/3/library/re.html).  In  EGREP mode, matching is case-insensitive
       when the query does not contain upper case characters, but is case-sensitive when it does.
       In  EGREPCS  mode, matching is always case-sensitive. Expressions will match anywhere in a
       metadata string, not only at the start.

       When multiple queries are given,  all  queries  have  to  match  for  a  search  hit  (AND
       behavior).

       It is possible to search individual metadata key/value items by prefixing the query with a
       metadata key name, separated by a colon  (':').  The  key  name  can  also  be  a  regular
       expression  to match multiple keys. A query match happens when any value of an item with a
       matching key name matches the query (OR behavior). See examples for more information.

       Examples:

       Query for (what happens to be) an author::

         % datalad search haxby

       Queries are case-INsensitive when the query contains no upper case characters, and can  be
       regular  expressions.  Use  EGREPCS  mode  when  it is desired to perform a case-sensitive
       lowercase match::

         % datalad search --mode egrepcs halchenko.*haxby

       This search mode performs NO analysis of the  metadata  content.   Therefore  queries  can
       easily  fail  to  match.  For example, the above query implicitly assumes that authors are
       listed in alphabetical order.  If that is the case (which may or may  not  be  true),  the
       following query would yield NO hits::

         % datalad search Haxby.*Halchenko

       The TEXTBLOB search mode represents an alternative that is more robust in such cases.

       For more complex queries multiple query expressions can be provided that all have to match
       to be considered a hit  (AND  behavior).  This  query  discovers  all  files  (non-default
       behavior) that match 'bids.type=T1w' AND 'nifti1.qform_code=scanner'::

         %    datalad   -c   datalad.search.index-egrep-documenttype=all   search   bids.type:T1w
       nifti1.qform_code:scanner

       Key name selectors can also be expressions, which can be used to select multiple  keys  or
       construct "fuzzy" queries. In such cases a query matches when any item with a matching key
       matches the query (OR behavior).  However, multiple queries are always evaluated using  an
       AND  conjunction.   The  following query extends the example above to match any files that
       have either 'nifti1.qform_code=scanner' or 'nifti1.sform_code=scanner'::

         %   datalad   -c   datalad.search.index-egrep-documenttype=all   search    bids.type:T1w
       nifti1.(q|s)form_code:scanner

   Mode: textblob
       This  search  mode  is  very  similar to the EGREP mode, but with a few key differences. A
       search index is built from the string-representation of metadata records. By default, only
       datasets  are included in this index, hence the indexing is usually completed within a few
       seconds, even for hundreds of datasets. This mode uses its own query language (not regular
       expressions) that is similar to other search engines. It supports logical conjunctions and
       fuzzy search terms. More information on this is available from the Whoosh project  (search
       engine implementation):

       - Description of the Whoosh query language:
         http://whoosh.readthedocs.io/en/latest/querylang.html)

       - Description of a number of query language customizations that are
         enabled in DataLad, such as, fuzzy term matching:
         http://whoosh.readthedocs.io/en/latest/parsing.html#common-customizations

       Importantly,  search  hits are scored and reported in order of descending relevance, hence
       limiting the number of search results is more meaningful than in the 'egrep' mode and  can
       also reduce the query duration.

       Examples:

       Search  for (what happens to be) two authors, regardless of the order in which those names
       appear in the metadata::

         % datalad search --mode textblob halchenko haxby

       Fuzzy search when you only have an approximate idea what you are looking for or how it  is
       spelled::

         % datalad search --mode textblob haxbi~

       Very  fuzzy  search,  when you are basically only confident about the first two characters
       and how it sounds approximately (or more precisely: allow  for  three  edits  and  require
       matching of the first two characters)::

         % datalad search --mode textblob haksbi~3/2

       Combine fuzzy search with logical constructs::

         % datalad search --mode textblob 'haxbi~ AND (hanke OR halchenko)'

   Mode: autofield
       This  mode  is  similar  to  the 'textblob' mode, but builds a vastly more detailed search
       index that represents individual metadata variables as individual fields. By default, this
       search  index  includes records for datasets and individual fields, hence it can grow very
       quickly into a huge structure that can easily take an hour or more to  build  and  require
       more  than  a  GB  of  storage.  However, limiting it to documents on datasets (see above)
       retains the enhanced expressiveness of queries while dramatically  reducing  the  resource
       demands.

       Examples:

       List names of search index fields (auto-discovered from the set of indexed datasets) which
       either have a field starting with "age" or "gender"::

         % datalad search --mode autofield --show-keys name '.age' '.gender'

       Fuzzy search for datasets with an author  that  is  specified  in  a  particular  metadata
       field::

         % datalad search --mode autofield bids.author:haxbi~ type:dataset

       Search  for  individual files that carry a particular description prefix in their 'nifti1'
       metadata::

         % datalad search --mode autofield nifti1.description:FSL* type:file

   Reporting
       Search  hits  are  returned  as  standard  DataLad  results.  On  the  command  line   the
       '--output-format' (or '-f') option can be used to tweak results for further processing.

       Examples:

       Format search hits as a JSON stream (one hit per line)::

         % datalad -f json search haxby

       Custom  formatting:  which  terms  matched  the  query  of  particular results. Useful for
       investigating fuzzy search results::

         $ datalad -f '{path}: {query_matched}' search --mode autofield bids.author:haxbi~

OPTIONS

       QUERY  query string, supported syntax and features depends on  the  selected  search  mode
              (see documentation).

       -h, --help, --help-np
              show  this  help  message.  --help-np  forcefully  disables  the use of a pager for
              displaying the help message

       -d DATASET, --dataset DATASET
              specify the dataset to perform the query operation on. If no dataset is  given,  an
              attempt  is  made  to  identify  the dataset based on the current working directory
              and/or the PATH given. Constraints: Value must be a Dataset or a  valid  identifier
              of a Dataset (e.g. a path)

       --reindex
              force  rebuilding  the  search  index, even if no change in the dataset's state has
              been detected, for example, when the index documenttype configuration has changed.

       --max-nresults MAX_NRESULTS
              maxmimum number of search results to report. Setting this  to  0  will  report  all
              search  matches. Depending on the mode this can search substantially slower. If not
              specified, a mode-specific default setting will be used. Constraints: value must be
              convertible to type 'int'

       --mode {egrep, textblob, autofield}
              Mode of search index structure and content. See section SEARCH MODES for details.

       --full-record, -f
              If  set,  return  the  full  metadata  record for each search hit. Depending on the
              search mode this might require additional queries. By default, only  data  that  is
              available  to  the  respective  search  modes  is  returned.  This  always includes
              essential information, such as the path and the type.

       --show-keys {name, short, full}
              if given, a list of known search keys is shown.  If  'name'  -  only  the  name  is
              printed  one  per line. If 'short' or 'full', statistics (in how many datasets, and
              how many unique values) are  printed.  'short'  truncates  the  listing  of  unique
              values.  QUERY,  if  provided,  is  regular  expressions  any  of which keys should
              contain. No other action is  performed  (except  for  reindexing),  even  if  other
              arguments  are  given.  Each key is accompanied by a term definition in parenthesis
              (TODO). In most cases a definition is given in the form of a URL.  If  an  ontology
              definition  for  a term is known, this URL can resolve to a webpage that provides a
              comprehensive definition of the term. However, for speed reasons term resolution is
              solely  done on information contained in a local dataset's metadata, and definition
              URLs might be outdated or point to no longer existing resources.

       --show-query
              if given, the formal query that was generated from the given query string is shown,
              but not actually executed. This is mostly useful for debugging purposes.

       --version
              show the module and its version which provides the command

AUTHORS

        datalad is developed by The DataLad Team and Contributors <team@datalad.org>.