NAME

       pegasus-exitcode - Checks the stdout/stderr files of a workflow job for any indication
       that an error occurred in the job. This script is intended to be invoked automatically by
       DAGMan as the POST script of a job.

SYNOPSIS

       pegasus-exitcode [-h][-t n][-r rv][-n] job.out

DESCRIPTION

       pegasus-exitcode is a utility that examines the STDOUT of a job to determine if the job
       failed, and renames the STDOUT and STDERR files of a job to preserve them in case the job
       is retried.

       Pegasus uses pegasus-exitcode as the DAGMan postscript for all jobs submitted via Globus
       GRAM. This tool exists as a workaround to a known problem with Globus where the exitcodes
       of GRAM jobs are not returned. This is a problem because Pegasus uses the exitcode of a
       job to determine if the job failed or not.

       In order to get around the exitcode problem, Pegasus wraps all GRAM jobs with Kickstart,
       which records the exitcode of the job in an XML invocation record, which it writes to the
       job’s STDOUT. The STDOUT is transferred from the execution host back to the submit host
       when the job terminates. After the job terminates, DAGMan runs the job’s postscript, which
       Pegasus sets to be pegasus-exitcode. pegasus-exitcode looks at the invocation record
       generated by kickstart to see if the job succeeded or failed. If the invocation record
       indicates a failure, then pegasus-exitcode returns a non-zero result, which indicates to
       DAGMan that the job has failed. If the invocation record indicates that the job succeeded,
       then pegasus-exitcode returns 0, which tells DAGMan that the job succeeded.

       pegasus-exitcode performs several checks to determine whether a job failed or not. These
       checks include:

        1. Is STDOUT empty? If it is empty, then the job failed.

        2. Are there any <status> tags with a non-zero value? If there are, then the job failed.
           Note that, if this is a clustered job, there could be multiple <status> tags, one for
           each task. If any of them are non-zero, then the job failed.

        3. Is there at least one <status> tag with a zero value? There must be at least one
           successful invocation or the job has failed.

       In addition, pegasus-exitcode allows the caller to specify the exitcode returned by Condor
       using the --return argument. This can be passed to pegasus-exitcode in a DAGMan post
       script by using the $RETURN variable. If this value is non-zero, then pegasus-exitcode
       returns a non-zero result before performing any other checks. For GRAM jobs, the value of
       $RETURN will always be 0 regardless of whether the job failed or not.

       Also, pegasus-exitcode allows the caller to specify the number of successful tasks it
       should see using the --tasks argument. If pegasus-exitcode does not see N successful
       tasks, where N is set by --tasks, then it will return a non-zero result. The default value
       is 1. This can be used to detect failures in clustered jobs where, for any number of
       reasons, invocation records do not get generated for all the tasks in the clustered job.

       In addition to checking the success/failure of a job, pegasus-exitcode also renames the
       STDOUT and STDERR files of the job so that if the job is retried, the STDOUT and STDERR of
       the previous run are not lost. It does this by appending a sequence number to the end of
       the files. For example, if the STDOUT file is called "job.out", then the first time the
       job is run pegasus-exitcode will rename the file "job.out.000". If the job is run again,
       then pegasus-exitcode sees that "job.out.000" already exists and renames the file
       "job.out.001". It will continue to rename the file by incrementing the sequence number
       every time the job is executed.

OPTIONS

       -h, --help
           Prints a usage summary with all the available command-line options.

       -t n, --tasks n
           Number of tasks expected. If less than n tasks succeeded, then pegasus-exitcode will
           fail with a non-zero return value. This is used in cases where we may not get a
           Kickstart invocation record for some tasks. Normally Seqexec will detect failed
           Kickstart invocations and fail accordingly.

       -r rv, --return rv
           Return value reported by DAGMan. This can be specified in the DAG using the $RETURN
           variable. If this is non-zero, then pegasus-exitcode immediately fails with a non-zero
           return value itself.

       -n, --no-rename
           Don’t rename job.out and job.err to .out.XXX and .err.XXX. This option is used
           primarily for testing.

AUTHORS

       Gideon Juve <juve@usc.edu>

       Pegasus Team http://pegasus.isi.edu

                                            02/28/2012                        PEGASUS-EXITCODE(1)