Provided by: datalad_0.12.4-2_all bug


       datalad search - search dataset metadata


       datalad  search  [-h]  [-d  DATASET]  [--reindex]  [--max-nresults  MAX_NRESULTS]  [--mode
              {egrep,textblob,autofield}]   [--full-record]    [--show-keys    {name,short,full}]
              [--show-query] [QUERY [QUERY ...]]


       DataLad can search metadata extracted from a dataset and/or aggregated into a superdataset
       (see the AGGREGATE-METADATA command). This makes it  possible  to  discover  datasets,  or
       individual files in a dataset even when they are not available locally.

       Ultimately  DataLad  metadata are a graph of linked data structures. However, this command
       does not (yet) support queries that can exploit all information stored in the metadata. At
       the  moment the following search modes are implemented that represent different trade-offs
       between the expressiveness of a query and the computational and storage resources required
       to execute a query.

       - egrep (default)

       - egrepcs [case-sensitive egrep]

       - textblob

       - autofield

       An  alternative  default  mode  can  be  configured  by  tuning the configuration variable

       [datalad "search"]
         default-mode = egrepcs

       Each search mode has its own default configuration for what kind of  documents  to  query.
       The respective default can be changed via configuration variables::

       [datalad "search"]
         index-<mode_name>-documenttype = (all|datasets|files)

   Mode: egrep/egrepcs
       These  search  modes  are  largely  ignorant of the metadata structure, and simply perform
       matching of a search pattern against a flat string-representation  of  metadata.  This  is
       advantageous  when  the  query  is  simple  and  the  metadata structure is irrelevant, or
       precisely known.  Moreover, it does not require a  search  index,  hence  results  can  be
       reported  without  an  initial  latency  for  building  a search index when the underlying
       metadata has changed (e.g. due to a dataset update). By default, these search  modes  only
       consider  datasets  and do not investigate records for individual files for speed reasons.
       Search results are reported in the order in which they were discovered.

       Queries     can     make     use     of     Python     regular      expression      syntax
       (  In  EGREP mode, matching is case-insensitive
       when the query does not contain upper case characters, but is case-sensitive when it does.
       In  EGREPCS  mode, matching is always case-sensitive. Expressions will match anywhere in a
       metadata string, not only at the start.

       When multiple queries are given,  all  queries  have  to  match  for  a  search  hit  (AND

       It is possible to search individual metadata key/value items by prefixing the query with a
       metadata key name, separated by a colon  (':').  The  key  name  can  also  be  a  regular
       expression  to match multiple keys. A query match happens when any value of an item with a
       matching key name matches the query (OR behavior). See examples for more information.


       Query for (what happens to be) an author::

         % datalad search haxby

       Queries are case-INsensitive when the query contains no upper case characters, and can  be
       regular  expressions.  Use  EGREPCS  mode  when  it is desired to perform a case-sensitive
       lowercase match::

         % datalad search --mode egrepcs halchenko.*haxby

       This search mode performs NO analysis of the  metadata  content.   Therefore  queries  can
       easily  fail  to  match.  For example, the above query implicitly assumes that authors are
       listed in alphabetical order.  If that is the case (which may or may  not  be  true),  the
       following query would yield NO hits::

         % datalad search Haxby.*Halchenko

       The TEXTBLOB search mode represents an alternative that is more robust in such cases.

       For more complex queries multiple query expressions can be provided that all have to match
       to be considered a hit  (AND  behavior).  This  query  discovers  all  files  (non-default
       behavior) that match 'bids.type=T1w' AND 'nifti1.qform_code=scanner'::

         %    datalad   -c   search   bids.type:T1w

       Key name selectors can also be expressions, which can be used to select multiple  keys  or
       construct "fuzzy" queries. In such cases a query matches when any item with a matching key
       matches the query (OR behavior).  However, multiple queries are always evaluated using  an
       AND  conjunction.   The  following query extends the example above to match any files that
       have either 'nifti1.qform_code=scanner' or 'nifti1.sform_code=scanner'::

         %   datalad   -c   search    bids.type:T1w

   Mode: textblob
       This  search  mode  is  very  similar to the EGREP mode, but with a few key differences. A
       search index is built from the string-representation of metadata records. By default, only
       datasets  are included in this index, hence the indexing is usually completed within a few
       seconds, even for hundreds of datasets. This mode uses its own query language (not regular
       expressions) that is similar to other search engines. It supports logical conjunctions and
       fuzzy search terms. More information on this is available from the Whoosh project  (search
       engine implementation):

       - Description of the Whoosh query language:

       - Description of a number of query language customizations that are
         enabled in DataLad, such as, fuzzy term matching:

       Importantly,  search  hits are scored and reported in order of descending relevance, hence
       limiting the number of search results is more meaningful than in the 'egrep' mode and  can
       also reduce the query duration.


       Search  for (what happens to be) two authors, regardless of the order in which those names
       appear in the metadata::

         % datalad search --mode textblob halchenko haxby

       Fuzzy search when you only have an approximate idea what you are looking for or how it  is

         % datalad search --mode textblob haxbi~

       Very  fuzzy  search,  when you are basically only confident about the first two characters
       and how it sounds approximately (or more precisely: allow  for  three  edits  and  require
       matching of the first two characters)::

         % datalad search --mode textblob haksbi~3/2

       Combine fuzzy search with logical constructs::

         % datalad search --mode textblob 'haxbi~ AND (hanke OR halchenko)'

   Mode: autofield
       This  mode  is  similar  to  the 'textblob' mode, but builds a vastly more detailed search
       index that represents individual metadata variables as individual fields. By default, this
       search  index  includes records for datasets and individual fields, hence it can grow very
       quickly into a huge structure that can easily take an hour or more to  build  and  require
       more  than  a  GB  of  storage.  However, limiting it to documents on datasets (see above)
       retains the enhanced expressiveness of queries while dramatically  reducing  the  resource


       List names of search index fields (auto-discovered from the set of indexed datasets)::

         % datalad search --mode autofield --show-keys name

       Fuzzy  search  for  datasets  with  an  author  that is specified in a particular metadata

         % datalad search --mode autofield type:dataset

       Search for individual files that carry a particular description prefix in  their  'nifti1'

         % datalad search --mode autofield nifti1.description:FSL* type:file

       Search   hits  are  returned  as  standard  DataLad  results.  On  the  command  line  the
       '--output-format' (or '-f') option can be used to tweak results for further processing.


       Format search hits as a JSON stream (one hit per line)::

         % datalad -f json search haxby

       Custom formatting: which terms  matched  the  query  of  particular  results.  Useful  for
       investigating fuzzy search results::

         $ datalad -f '{path}: {query_matched}' search --mode autofield


       QUERY  query  string,  supported  syntax  and features depends on the selected search mode
              (see documentation).

       -h, -\-help, -\-help-np
              show this help message. --help-np forcefully  disables  the  use  of  a  pager  for
              displaying the help message

       -d DATASET, -\-dataset DATASET
              specify  the  dataset to perform the query operation on. If no dataset is given, an
              attempt is made to identify the dataset based  on  the  current  working  directory
              and/or  the  PATH given. Constraints: Value must be a Dataset or a valid identifier
              of a Dataset (e.g. a path)

              force rebuilding the search index, even if no change in  the  dataset's  state  has
              been detected, for example, when the index documenttype configuration has changed.

       -\-max-nresults MAX_NRESULTS
              maxmimum  number  of  search  results  to report. Setting this to 0 will report all
              search matches. Depending on the mode this can search substantially slower. If  not
              specified, a mode-specific default setting will be used. Constraints: value must be
              convertible to type 'int'

       -\-mode {egrep, textblob, autofield}
              Mode of search index structure and content. See section SEARCH MODES for details.

       -\-full-record, -f
              If set, return the full metadata record for  each  search  hit.  Depending  on  the
              search  mode  this  might require additional queries. By default, only data that is
              available to  the  respective  search  modes  is  returned.  This  always  includes
              essential information, such as the path and the type.

       -\-show-keys {name, short, full}
              if  given,  a  list  of  known  search  keys is shown. If 'name' - only the name is
              printed one per line. If 'short' or 'full', statistics (in how many  datasets,  and
              how  many  unique  values)  are  printed.  'short'  truncates the listing of unique
              values. No other action  is  performed  (except  for  reindexing),  even  if  other
              arguments  are  given.  Each key is accompanied by a term definition in parenthesis
              (TODO). In most cases a definition is given in the form of a URL.  If  an  ontology
              definition  for  a term is known, this URL can resolve to a webpage that provides a
              comprehensive definition of the term. However, for speed reasons term resolution is
              solely  done on information contained in a local dataset's metadata, and definition
              URLs might be outdated or point to no longer existing resources.

              if given, the formal query that was generated from the given query string is shown,
              but not actually executed. This is mostly useful for debugging purposes.


        datalad is developed by The DataLad Team and Contributors <>.