NAME

     nq — job queue utility

SYNOPSIS

     nq [-q] command line ...
     nq -t job id ...
     nq -w job id ...

DESCRIPTION

     The nq utility provides a very lightweight queuing system without requiring setup,
     maintenance, supervision or any long-running processes.

     Job order is enforced by a timestamp nq gets immediately when started.  Synchronization
     happens on file-system level.  Timer resolution is milliseconds.  No sub-second file system
     time stamps are required.  Polling is not used.  Exclusive execution is maintained strictly.

     You enqueue(!) new jobs into the queue by running

           nq command line ...

     The job id (a file name relative to NQDIR, which defaults to the current directory) is
     output (unless suppressed using -q) and nq detaches from the terminal immediately, running
     the job in the background.  Standard output and standard error are redirected into the job
     id file.  fq(1) can be used to conveniently watch the log files.

     The options are as follows:

     -q      Suppress output of the job id after spawning new job.

     -t      Enter test mode: exit with status 0 when all of the listed job ids are already done,
             else with status 1.

     -w      Enter waiting mode: wait in the foreground until all listed job ids are done.

ENVIRONMENT

     NQDIR   Directory where lock files/job output resides.  Each NQDIR can be considered a
             separate queue.  The current working directory is used when NQDIR is unset.  NQDIR
             is created if needed.

     NQJOBID The job id of the currently running job, exposed to the job itself.

FILES

     nq owns all files in NQDIR (respectively .) which start with “,” or “.,”.  These files are
     created according to the following scheme:

           ,hexadecimal-time-stamp.pid

EXIT STATUS

     The nq utility exits 0 on success, and >0 if an error occurs; unless test mode is used, in
     which case exit status 1 means there is a job running.

     On fatal errors, exit codes 111 and 222 are used.

EXAMPLES

     Build make(1) targets clean, depends, all, without occupying the terminal:

           % nq make clean
           % nq make depends
           % nq make all
           % fq
           ... look at output, can interrupt with C-c any time
           without stopping the build ...

     Simple download queue, accessible from multiple terminals:

           % alias qget='NQDIR=/tmp/downloads nq wget'
           % alias qwait='NQDIR=/tmp/downloads fq -q'
           window1% qget http://mymirror/big1.iso
           window2% qget http://mymirror/big2.iso
           window3% qget http://mymirror/big3.iso
           % qwait
           ... wait for all downloads to finish ...

     As nohup(1) replacement (The benchmark will run in background, every run gets a different
     output file, and the command line you ran is logged too.):

           % ssh remote
           remote% nq ./run-benchmark
           ,14f6f3034f8.17035
           remote% ^D
           % ssh remote
           remote% fq
           ... see output, fq exits when job finished ...

TRICKS

     The "file extension" of the log file is actually the PID of the job.  nq runs all jobs in a
     separate process group, so you can kill an entire job process tree at once using kill(1)
     with a negative PID.  Before the job is started, it is the PID of nq, so you can cancel a
     queued job by killing it as well.

     Thanks to the initial exec line in the log files, you can resubmit a job by executing it as
     a shell command file, i.e. running

           sh job id

     Creating nq wrappers setting NQDIR to provide different queues for different purposes is
     encouraged.

INTERNALS

     Enforcing job order works like this:
     -   every job has an output file locked using flock(2) and named according to FILES.
     -   every job starts only after all earlier flocked files are unlocked.
     -   the lock is released by the kernel after the job terminates.

ASSUMPTIONS

     nq will only work correctly when:

     -   NQDIR (respectively .) is writable.

     -   flock(2) works correctly in NQDIR (respectively .).

     -   gettimeofday(2) behaves monotonic (using CLOCK_MONOTONIC would create confusing file
         names after reboot).

     -   No other programs put files matching ,* into NQDIR (respectively .).

SEE ALSO

     fq(1), tq(1).

     Alternatives to the nq system include batch(1), qsub(1), schedule(1), srun(1), and ts(1).

AUTHORS

     Leah Neukirchen <leah@vuxu.org>

CAVEATS

     All reliable queue status information is in main memory only, which makes restarting a job
     queue after a reboot difficult.

LICENSE

     nq is in the public domain.

     To the extent possible under law, the creator of this work has waived all copyright and
     related or neighboring rights to this work.

     http://creativecommons.org/publicdomain/zero/1.0/