NAME

       grcollect - performing transposition on the input tabulated data

SYNOPSIS

       grcollect [options] <input> [...] [-o <output>|-b <basename>]

DESCRIPTION

       The  main  purpose of the program `grcollect` is twofold. First, it is intended to do data
       transposition on the input data, i.e. the input (which is  read  from  files  or  standard
       input)  is  sorted  and  splitted  to  separate  files  where  the splitting is based on a
       respective key. These keys are taken from the input data. In such a case where  the  input
       is  from  more  files  and each key is unique in a given file, this process is called data
       transposition (since it is similar when a 2 dimensional data matrix is stored in the  form
       as  each  row  is  in a separate file, and one intends to transpose the matrix, i.e. store
       each column in a separate file). The other feature of `grcollect` is to do  some  sort  of
       statistics  on  data associated to different keys. These statistics include average (mean,
       median, mode) and scatter (standard deviation or median  deviance)  estimations  with  the
       optional deselection of outlier points, summation, count statistics and so on.

OPTIONS

   General options:
       -h, --help
              Give general summary about the command line options.

       --long-help, --help-long
              Gives a detailed list of command line options.

       --wiki-help, --help-wiki, --mediawiki-help, --help-mediawiki
              Gives a detailed list of command line options in Mediawiki format.

       --version, --version-short, --short-version
              Give some version information about the program.

       <input> [,<input>, ...]
              Name  of  the  input  file.  At  least,  one file should be specified. Reading from
              standard input can be forced using a single dash  "-"  as  input  file  name.  More
              dashes are silently ignored.

       -c, --col-base <key column index>
              Column index for the key.

   Data transposition specific options:
       -b, --basename <base-%b-name>
              Base  name  of  the output files. The base name string should conatain at least one
              "%b" tag, which is replaced by the respective key string on  the  creation  of  the
              file.

       -x, --extension <extension>, -p, --prefix <prefix>
              Equivalent  to  "-b|--basename  <prefix>%b.<extension>".  Note  that  in  practice,
              <prefix> might be some sort of directory name  and  extension  is  a  regular  file
              extension,  but  the  above  substitution  is  done literally. Therefore, the "dot"
              between the key and the <extension> is always inserted in the  final  name  of  the
              output  files  but a trailing slash is required at the end of <prefix> if the files
              are to be created in that particular directory.  Note  also  that  this  case,  the
              target  directory  must  exist  before the invocation of `grcollect`, otherwise the
              output files cannot be created.

       -C, --comment
              Insert a commented line  (starting  with  "#")  containing  information  about  the
              version  and  command line invocation syntax of `grcollect` to the beginning of the
              transposed files.

       -S, --additional-comment <...>
              Insert an additional commented lines (starting with "#") to the  beginning  of  the
              transposed files.

   Options for cumulative statistics:
       -d, --col-stat <>[,...]
              Comma-separated  list  of  column  indices  on  which  the  statistics  are  to  be
              calculated. Columns with non-numerical contents are ignored.Note that  this  option
              imply the cumulative statistics mode of `grcollect`.

       -o, --output <filename>
              The  name  of the output file to which the output statistics are written. The total
              number of columns in this file will be 1+C*N, where C is the number of columns (see
              -d|--col-stat)  on  which  the  statistics  are  calculated  and N is the number of
              statistic quantities (see --stat). The first column in the output file is the  key,
              which  is  followed  by the per-column list of statistics, in the same order as the
              user defined after -d|--col-stat and --stat.

       -s, --stat <list of statistics>
              Comma-separated list of statistics to be estimated on the input data. These can  be
              one or more of the following:

       count  Total number of records, for the given key.

       rcount The  number  of records after rejecting outliers (i.e. it is always the same as the
              "count" value if no "--rejection" was used).

       mean, median, mode
              Mean, median or mode statistics of the data.

       rmean, rmedian, rmode
              Mean, median or mode, after rejecting outliers.

       {mean|median|mode}stddev, {mean|median|mode}meddev, stddev
              Scatter of the data around the mean, median or mode.  The  scatter  can  either  be
              standard  deviation  (stddev)  or median deviance (meddev). The literal "stddev" is
              the classic standard deviation, equivalent to "meanstddev".

       r{mean|median|mode}stddev, r{mean|median|mode}meddev, rstddev
              The same scatters as above but after rejecting outliers.

       sum, rsum
              Sum of the data, esp. total sum and sum after rejecting outliers.

       sum2, rsum2
              Sum of the squares, total and after rejecting outliers.

       min, max
              Minimal and maximal data values.

       rmin, rmax
              Minimal and maximal data values after the rejection of outliers.

       -r, --rejection column=<index>,<rejection parameters>
              Comma-separated directives for outlier rejection  for  the  specified  column.  The
              rejection parameters are:

       iterations=<n>
              Maximum number of iterations to reject outliers.

       mean, median, mode
              Use the mean, median or mode for the center of the rejection.

       stddev, meddev, absolute=<limit>
              Use  the standard deviation or median deviance  for rejection limit units or define
              an absolute limit for rejection level.

       Note that each column can have different kind of rejection  method,  thus  more  than  one
       "--rejection ..." command line option can be used at the invocation of `grcollect`.

   Other options:
       -m, --max-memory <memory>[kmg]
              Maximum  amount  of  memory available for `grcollect`. The prefixes "k", "m" or "g"
              can be used for kilobytes, megabytes and gigabytes, respectively. On 32bit systems,
              the maximum memory is limited to 3gigabytes. Note that `grcollect` does not use any
              kind of operating system specific  methods  to  determine  the  maximum  amount  of
              memory,  it  always  should be set by the user. The default value of 8 megabytes is
              somewhat small, so upon massive data transposition (tens or hundreds of gigabytes),
              this limit is worth to be set accordingly to the physical memory available.

       -t, --tmpdir <directory>
              Directory  for temporary file storage. Note that the default temporary directory is
              always the current one (which is is equivalent to define "--tmpdir ./"), since in a
              usual  configuration  the /tmp directory is small, moreover, it can be some sort of
              "tmpfs", temporary file system mount on the physical memory itself.

REPORTING BUGS

       Report bugs to <apal@szofi.net>, see also https://fitsh.net/.

COPYRIGHT

       Copyright © 1996, 2002, 2004-2008, 2010-2016, 2018-2020; Pal, Andras <apal@szofi.net>