NAME

       allpairs_master - executes All-Pairs workflow in parallel on distributed systems

SYNOPSIS

       allparis_master [options] <set A> <set B> <compare function>

DESCRIPTION

       allpairs_master  computes  the  Cartesian  product  of  two  sets  (<set  A> and <set B>),
       generating a matrix where each cell M[i,j] contains the output of the function F (<compare
       function>)  on  objects  A[i]  (an  item  in  <set  A>) and B[j] (an item in <set B>). The
       resulting matrix is displayed on the standard output, one comparison result per line along
       with the associated X and Y indices.

       allpairs_master  uses  the  Work  Queue system to distribute tasks among processors.  Each
       processor utilizes the allpairs_multicore(1) program to execute the tasks in  parallel  if
       multiple  cores  are  present.  After starting allpairs_master, you must start a number of
       work_queue_worker(1) processes on remote machines.  The workers will then connect back  to
       the master process and begin executing tasks.

OPTIONS

       -p, --port=<port>
              The port that the master will be listening on.

       -e, --extra-args=<args>
              Extra arguments to pass to the comparison function.

       -f, --input-file=<file>
              Extra input file needed by the comparison function. (may be given multiple times)

       -o, --debug-file=<file>
              Write  debugging  output  to  this  file.  By  default, debugging is sent to stderr
              (":stderr"). You may specify logs be sent to  stdout  (":stdout"),  to  the  system
              syslog (":syslog"), or to the systemd journal (":journal").

       -O, ----output-file=<file>
              Write task output to this file (default to standard output)

       -t, --estimated-time=<seconds>
              Estimated time to run one comparison. (default chosen at runtime)

       -x, --width=<item>
              Width of one work unit, in items to compare. (default chosen at runtime)

       -y, --height=<items>
              Height of one work unit, in items to compare. (default chosen at runtime)

       -N, --project-name=<project>
              Report the master information to a catalog server with the project name - <project>

       -P, --priority=<integer>
              Priority. Higher the value, higher the priority.

       -d, --debug=<flag>
              Enable debugging for this subsystem. (Try -d all to start.)

        -v, --version
              Show program version.

        -h,--help <>
              Display this message.

       -Z, --port-file=<file>
              Select port at random and write it to this file.  (default is disabled)

        --work-queue-preferred-connection <connection>
              Indicate  preferred  connection.  Chose  one  of  by_ip or by_hostname. (default is
              by_ip)

EXIT STATUS

       On success, returns zero.  On failure, returns non-zero.

EXAMPLES

       Let's suppose you have a whole lot of files that you want to compare all  to  each  other,
       named  a,  b, c, and so on. Suppose that you also have a program named compareit that when
       invoked as compareit a b will compare files a and b and produce  some  output  summarizing
       the difference between the two, like this:

                a b are 45 percent similar

       To  use  the  allpairs  framework,  create  a file called set.list that lists each of your
       files, one per line:

                a
                b
                c
                ...

       Because   allpairs_master   utilizes   allpairs_multicore(1),   so   please   make    sure
       allpairs_multicore(1)  is  in  your  PATH  before  you proceed.To run a All-Pairs workflow
       sequentially, start a single work_queue_worker(1) process in the background. Then,  invoke
       allpairs_master.

                % work_queue_worker localhost 9123 &
                % allpairs_master set.list set.list compareit

       The  framework  will  carry  out  all  possible  comparisons of the objects, and print the
       results one by one (note that the first two columns are X and Y indices in  the  resulting
       matrix):

                1   1    a a are 100 percent similar
                1   2    a b are 45 percent similar
                1   3    a c are 37 percent similar
                ...

       To speed up the process, run more work_queue_worker(1) processes on other machines, or use
       condor_submit_workers(1) or sge_submit_workers(1) to start hundreds  of  workers  in  your
       local batch system.

       The  following  is  an  example  of  adding  more workers to execute a All-Pairs workflow.
       Suppose your allpairs_master is running on a machine  named  barney.nd.edu.  If  you  have
       access  to  login  to other machines, you could simply start worker processes on each one,
       like this:

                % work_queue_worker barney.nd.edu 9123

       If you have access to a batch system like Condor, you can submit multiple workers at once:

                % condor_submit_workers barney.nd.edu 9123 10
                Submitting job(s)..........
                Logging submit event(s)..........
                10 job(s) submitted to cluster 298.

COPYRIGHT

       The Cooperative Computing Tools are Copyright (C) 2003-2004 Douglas  Thain  and  Copyright
       (C)  2005-2015  The  University of Notre Dame.  This software is distributed under the GNU
       General Public License.  See the file COPYING for details.

SEE ALSO

       •   The Cooperative Computing Tools ("http://ccl.cse.nd.edu/software/manuals")

       •   All-Pairs User Manual ("http://ccl.cse.nd.edu/software/manuals/allpairs.html")

       •   Work Queue User Manual ("http://ccl.cse.nd.edu/software/manuals/workqueue.html")

       •   work_queue_worker(1)

       •   condor_submit_workers(1)

       •   sge_submit_workers(1)

       •   allpairs_multicore(1)