Ubuntu Manpage: stilts-cdsskymatch - Crossmatches table on sky position against VizieR/SIMBAD table

NAME

       stilts-cdsskymatch - Crossmatches table on sky position against VizieR/SIMBAD table

SYNOPSIS

       stilts cdsskymatch [ifmt=<in-format>] [istream=true|false] [in=<table>] [icmd=<cmds>]
                          [ocmd=<cmds>]
                          [omode=out|meta|stats|count|cgi|discard|topcat|samp|tosql|gui]
                          [out=<out-table>] [ofmt=<out-format>] [ra=<expr>] [dec=<expr>]
                          [radius=<value/arcsec>] [cdstable=<value>] [find=all|best|best-
                          remote|each|each-dist] [blocksize=<int-value>] [maxrec=<int-value>]
                          [compress=true|false] [serviceurl=<url-value>] [usemoc=true|false]
                          [presort=true|false] [fixcols=none|dups|all] [suffixin=<label>]
                          [suffixremote=<label>]

DESCRIPTION

cdsskymatch uses the CDS X-Match service to join a local table to one of the tables hosted
by the Centre de Données astronomiques de Strasbourg. This includes all of the VizieR
tables and the SIMBAD database. The service is very fast, and in most cases it is the best
way to match a local table against a large external table hosted by a service. It is
almost certainly much better than using coneskymatch, though it is less flexible than TAP
(see the tapquery task for flexible access to TAP services, and tapskymatch for positional
matches).

The local table is uploaded to the X-Match service in chunks, and the matches for each
chunk are retrieved in turn and eventually stitched together to form the final result. The
tool only uploads sky position and an identifier for each row of the input table, but all
columns of the input table are reinstated in the result for reference.

The remote table in most cases contains only a subset of the the columns in the relevant
VizieR table, including the most useful ones. The service currently provides no
straightforward way to acquire columns which are not returned by default.

Acknowledgement: CDS note that if the use of the X-Match service is useful to your
research, they would appreciate the following acknowledgement: "This research made use of
the cross-match service provided by CDS, Strasbourg."

OPTIONS

ifmt=<in-format>
Specifies the format of the input table as specified by parameter in. The known
formats are listed in SUN/256. This flag can be used if you know what format your
table is in. If it has the special value (auto) (the default), then an attempt will
be made to detect the format of the table automatically. This cannot always be done
correctly however, in which case the program will exit with an error explaining
which formats were attempted.

istream=true|false
If set true, the input table specified by the in parameter will be read as a
stream. It is necessary to give the ifmt parameter in this case. Depending on the
required operations and processing mode, this may cause the read to fail (sometimes
it is necessary to read the table more than once). It is not normally necessary to
set this flag; in most cases the data will be streamed automatically if that is the
best thing to do. However it can sometimes result in less resource usage when
processing large files in certain formats (such as VOTable).

in=<table>
The location of the input table. This may take one of the following forms:

* A filename.

* A URL.

* The special value "-", meaning standard input. In this case the input format
must be given explicitly using the ifmt parameter. Note that not all formats
can be streamed in this way.

* A system command line with either a "<" character at the start, or a "|"
character at the end ("<syscmd" or "syscmd|"). This executes the given pipeline
and reads from its standard output. This will probably only work on unix-like
systems.
In any case, compressed data in one of the supported compression formats (gzip,
Unix compress or bzip2) will be decompressed transparently.

icmd=<cmds>
Specifies processing to be performed on the input table as specified by parameter
in, before any other processing has taken place. The value of this parameter is one
or more of the filter commands described in SUN/256. If more than one is given,
they must be separated by semicolon characters (";"). This parameter can be
repeated multiple times on the same command line to build up a list of processing
steps. The sequence of commands given in this way defines the processing pipeline
which is performed on the table.

Commands may alteratively be supplied in an external file, by using the indirection
character '@'. Thus a value of "@filename" causes the file filename to be read for
a list of filter commands to execute. The commands in the file may be separated by
newline characters and/or semicolons.

ocmd=<cmds>
Specifies processing to be performed on the output table, after all other
processing has taken place. The value of this parameter is one or more of the
filter commands described in SUN/256. If more than one is given, they must be
separated by semicolon characters (";"). This parameter can be repeated multiple
times on the same command line to build up a list of processing steps. The sequence
of commands given in this way defines the processing pipeline which is performed on
the table.

omode=out|meta|stats|count|cgi|discard|topcat|samp|tosql|gui
The mode in which the result table will be output. The default mode is out, which
means that the result will be written as a new table to disk or elsewhere, as
determined by the out and ofmt parameters. However, there are other possibilities,
which correspond to uses to which a table can be put other than outputting it, such
as displaying metadata, calculating statistics, or populating a table in an SQL
database. For some values of this parameter, additional parameters (<mode-args>)
are required to determine the exact behaviour.

Possible values are

* out

* meta

* stats

* count

* cgi

* discard

* topcat

* samp

* tosql

* gui
Use the help=omode flag or see SUN/256 for more information.

out=<out-table>
The location of the output table. This is usually a filename to write to. If it is
equal to the special value "-" (the default) the output table will be written to
standard output.

This parameter must only be given if omode has its default value of "out".

ofmt=<out-format>
Specifies the format in which the output table will be written (one of the ones in
SUN/256 - matching is case-insensitive and you can use just the first few letters).
If it has the special value "(auto)" (the default), then the output filename will
be examined to try to guess what sort of file is required usually by looking at the
extension. If it's not obvious from the filename what output format is intended, an
error will result.

This parameter must only be given if omode has its default value of "out".

ra=<expr>
Right ascension in degrees in the ICRS coordinate system for the position of each
row of the input table. This may simply be a column name, or it may be an algebraic
expression calculated from columns as explained in SUN/256. If left blank, an
attempt is made to guess from UCDs, column names and unit annotations what
expression to use.

dec=<expr>
Declination in degrees in the ICRS coordinate system for the position of each row
of the input table. This may simply be a column name, or it may be an algebraic
expression calculated from columns as explained in SUN/256. If left blank, an
attempt is made to guess from UCDs, column names and unit annotations what
expression to use.

radius=<value/arcsec>
Maximum distance from the local table (ra,dec) position at which counterparts from
the remote table will be identified. This is a fixed value given in arcseconds, and
must be in the range [0,180] (this limit is currently enforced by the CDS Xmatch
service).

cdstable=<value>
Identifier of the table from the CDS crossmatch service that is to be matched
against the local table. This identifier may be the standard VizieR identifier
(e.g. "II/246/out" for the 2MASS Point Source Catalogue) or "simbad" to indicate
SIMBAD data.

See for instance the TAPVizieR table searching facility at http://tapvizier.u-
strasbg.fr/adql/ to find VizieR catalogue identifiers.

find=all|best|best-remote|each|each-dist
Determines which pair matches are included in the result.

* all: All matches

* best: Matched rows, best remote row for each input row

* best-remote: Matched rows, best input row for each remote row

* each: One row per input row, contains best remote match or blank

* each-dist: One row per input row, column giving distance only for best match
Note only the all mode is symmetric between the two tables.

Note also that there is a bug in best-remote matching. If the match is done in
multiple blocks, it's possible for a remote table row to appear matched against one
local table row per uploaded block, rather than just once for the whole result. If
you're worried about that, set blocksize >= rowCount. This may be fixed in a future
release.

blocksize=<int-value>
The CDS Xmatch service operates limits on the maximum number of rows that can be
uploaded and the maximum number of rows that is returned as a result from a single
query. In the case of large input tables, they are broken down into smaller blocks,
and one request is sent to the external service for each block. This parameter
controls the number of rows in each block. For an input table with fewer rows than
this value, the whole thing is done as a single request.

At time of writing, the maximum upload size is 100Mb (about 3Mrow; this does not
depend on the width of your table), and the maximum return size is 2Mrow.

Large blocksizes tend to be good (up to a point) for reducing the total amount of
time a large xmatch operation takes, but they can make it harder to see the job
progressing. There is also the danger (for ALL-type find modes) of exceeding the
return size limit, which will result in truncation of the returned result.

maxrec=<int-value>
Limit to the number of rows resulting from this operation. If the value is negative
(the default) no limit is imposed. Note however that there can be truncation of the
result if the number of records returned from a single chunk exceeds the service
hard limit (2,000,000 at time of writing).

compress=true|false
If true, the service is requested to provide HTTP-level compression for the
response stream (Accept-Encoding header is set to "gzip", see RFC 2616). This does
not guarantee that compression will happen but if the service honours this request
it may result in a smaller amount of network traffic at the expense of more
processing on the server and client.

serviceurl=<url-value>
The URL at which the CDS Xmatch service can be found. Normally this should not be
altered from the default, but if other implementations of the same service are
known, this parameter can be used to access them.

usemoc=true|false
If true, first acquire a MOC coverage map from CDS, and use that to pre-filter rows
before uploading them for matching. This should improve efficiency, but have no
effect on the result.

presort=true|false
If true, the rows are sorted by HEALPix index before they are uploaded to the CDS
X-Match service. If the match is done in multiple blocks, this may improve
efficiency, since when matching against a large remote catalogue the X-Match
service likes to process requests in which sources are grouped into a small region
rather than scattered all over the sky.

Note this will have a couple of other side effects that may be undesirable: it will
read all the input rows into the task at once, which may make it harder to assess
progress, and it will affect the order of the rows in the output table.

It is probably only worth setting true for rather large (multi-million-row?) multi-
block matches, where both local and remote catalogues are spread over a significant
fraction of the sky. But feel free to experiment.

fixcols=none|dups|all
Determines how input columns are renamed before use in the output table. The
choices are:

* none: columns are not renamed

* dups: columns which would otherwise have duplicate names in the output will be
renamed to indicate which table they came from

* all: all columns will be renamed to indicate which table they came from
If columns are renamed, the new ones are determined by suffix* parameters.

suffixin=<label>
If the fixcols parameter is set so that input columns are renamed for insertion
into the output table, this parameter determines how the renaming is done. It gives
a suffix which is appended to all renamed columns from the input table.

suffixremote=<label>
If the fixcols parameter is set so that input columns are renamed for insertion
into the output table, this parameter determines how the renaming is done. It gives
a suffix which is appended to all renamed columns from the CDS result table.

VERSION

       STILTS version 3.1-2-debian

       This is the Debian version of Stilts, which lack the support  of  some  file  formats  and
       network protocols. For differences see
       file:///usr/share/doc/stilts/README.Debian

AUTHOR