Provided by: filespooler_1.2.3-1build1_amd64 
NAME
fspl - sequential, distributed job queue processing
SYNOPSIS
fspl [ OPTIONS ] COMMAND [ command_options ]
OVERVIEW
fspl is the CLI part of the Filespooler (https://www.complete.org/filespooler) package.
fspl is a Unix-style tool that facilitates local or remote command execution, complete
with stdin capture, with easy integration with various tools. Here's a brief Filespooler
feature list:
• It can easily use tools such as S3, Dropbox, Syncthing, NNCP, ssh, UUCP, USB drives,
CDs, etc. as transport.
• Translation: you can use basically anything that is a filesystem as a transport
• It can use arbitrary decoder command pipelines (eg, zcat, stdcat, gpg, age, etc) to pre-
process stored packets.
• It can send and receive packets by pipes.
• Its storage format is simple on-disk files with locking.
• It supports one-to-one and one-to-many configurations.
• Locking is unnecessary when writing new jobs to the queue, and many arbitrary tools (eg,
Syncthing, Dropbox, etc) can safely write directly to the queue without any assistance.
• Queue processing is strictly ordered based on the order on the creation machine, even if
job files are delivered out of order to the destination.
• stdin can be piped into the job creation tool, and piped to a later executor at process
time on a remote machine.
• The file format is lightweight; less than 100 bytes overhead unless large extra
parameters are given.
• The queue format is lightweight; having 1000 different queues on a Raspberry Pi would be
easy.
• Processing is stream-based throughout; arbitrarily-large packets are fine and sizes in
the TB range are no problem.
• The Filespooler command, fspl, is extremely lightweight, consuming less than 10MB of RAM
on x86_64.
• Filespooler has extensive documentation.
Filespooler consists of a command-line tool (fspl) for interacting with queues. It also
consists of a Rust library that is used by fspl. main.rs for fspl is just a few lines
long.
A WORD ABOUT DOCUMENTATION
This manual is the reference for fspl. The filespooler homepage,
<https://www.complete.org/filespooler/> contains many examples, instructions on how to
integrate with everything from file syncers to encryption tools, and so forth. Please
refer to it for further information.
BASIC OPERATION
The basic idea is this:
• Before starting, on the receiving end, you run fspl queue-init to prepare a queue
directory.
• On the sending end, you use fspl prepare to prepare a job file (packet). This packet is
written to stdout. From there, you can pipe it to fspl write to inject it into a local
queue, or use various kinds of transport to get it to a remote machine.
• You use fspl queue-process to execute packets.
• Alternatively, the fspl stdin- series of commands let you have more manual control
over queue processing, accepting job packets in stdin. They can let you completely
ignore the built-in queue mechanism if you so desire.
ON-DISK FORMATS
The key way to ensure the ordered processing of the job queue is with a sequence number.
This is a 64-bit unsigned integer. It is stored in a seqfile on both the sending and the
receiving side. On the sending side, the seqfile is standalone; there is only an
accompanying .lock file for it. On the receiving side, the seqfile and its accompanying
lock file live within the queue directory.
When the seqfile is referenced on the sending side, it will be created and initialized
with the value 1 if it does not already exist. On the receiving side, it is created as
part of fspl queue-init.
In either case, the seqfile consists of one newline-terminated line, containing the next
number to process. On the sending side, this is used by fspl prepare as the sequence
number for the next generated packet. On the receiving side, it is used by fspl queue-
process to determine which job to process next (unless changed by --order-by).
THE QUEUE
The queue has this general layout:
queuedir/ Top-level queue directory
nextseq Sequence file
nextseq.lock Lock file
jobs/ Job files stored here
When passing the --queuedir to one of the fspl queue- commands, you give it the path to
the top-level queuedir as shown here.
You are free to create additional directories within the queuedir so long as they don't
use one of the names listed above. This can be helpful for receiving queue contents in
certain situations.
Append-Only Queues
You can specify --append-only to fspl queue-init, which will cause the nextseq and
nextseq.lock files to be omitted. This has the effect of making the queue write-only.
This can be useful if you are synchronizing the jobs subdirectory between machines, but
still want to be able to use fspl queue-write to add jobs to that folder. It will prevent
fspl queue-process from running. You can still inspect an append-only queue with commands
like fspl queue-ls and fspl queue-info.
JOB FILES
Job files live within queuedir/jobs. They all must follow this naming pattern:
fspl-*.fspl
This pattern is specifically designed to facilitate safe injection of job files into the
queue by other tools. Many other tools prepend or append a temporary string to a filename
to signify that it has not yet been fully transferred. The Filespooler assumption is that
once a file appears in jobs/ with a name matching this pattern, than it has been fully
transferred and can be processed at any time.
So long as the filename begins with fspl- and ends with .fspl, you are free to put
whatever string you like in the middle. The only other requirement, of course, is that
each job must have a unique filename within the directory. To simplify things, you can
pipe a job file to fspl queue-write and let that command take care of naming. Or, you can
generate a random (or non-random) string yourself in a shell script.
The job file itself consists of a small binary header, which is CRC32-checked. This
header is normally less than 100 bytes and the length of it is encoded within the file.
Following the header, if --input was given to fspl prepare, whatever was piped to prepare
is included as the "payload". This will be piped to the executor command when run by fspl
queue-process or fspl stdin-process. The payload is not validated by CRC or length by
Filespooler, since this is assumed to be the role of the transport layer. The website
contains examples of using GPG or other tools to ensure integrity.
There are three types of job files:
• Command, created by fspl prepare. This is the typical kind of job file, and is used to
request the execution of a command by the processor.
• NOP, created by fspl prepare-nop. This is a "no-op" job file, which does not run a
command but is considered to always succeed.
• Fail, created by fspl prepare-fail. This is a "fail" job file, which does not run a
command but is considered to always fail. This could be usedful, for instance, to
create a "barrier" to prevent a queue processor from continuing to execute commands past
there without human intervention.
ADDING FILES TO THE QUEUE
To expand slightly on the discussion above about adding files to the queue:
A common way to do this if your transport tool doesn't use a nice temporary name is to
transport the file to an adjacent directory, and then use mv(1) or, better, make a hard
link with ln(1) to get the file into the jobs/ directory. Note that in both cases, you
must take care that you are not crossing a filesystem boundary; on some platforms such as
Linux, mv will revert to copy instead of rename if you cross the boundary and then the
assumptions about completeness are violated.
JOB FILE ENCODING AND DECODING
Job files are, by default, stored exactly as laid out above. However, in many cases, it
may be desirable to store them "encoded" - compressed or encrypted. In this case, the
output from fspl prepare can be piped through, say, gzip and the resulting packet can
still be stored in jobs/ by fspl queue-write or any related tool.
Now, however, we arrive at the question: how can Filespooler process a queue containing
files that have been compressed, encrypted, or so forth?
Every fspl queue command takes an optional --decoder (or -d) parameter, which is a command
string that will be executed by the shell. This decoder command will receive the entire
job file (not just the payload) piped to it on stdin, and is expected to write the decoded
file to stdout.
The fspl stdin pairs to the queue commands do not accept a decoder parameter, since it is
assumed you would do that in the pipeline on the way to the stdin command.
For instance:
date | fspl prepare -s ~/state -i - | gzip | fspl queue-write -q ~/queue
fspl queue-ls -q ~/queue -d zcat
ID creation timestamp filename
48 2022-05-07T21:07:02-05:00 fspl-48aa52ad-c65c-478a-9d37-123d4bebcb30.fspl
Normally, fspl ignores files that fail to decode the header. If you omit the --decoder,
it may just look like your queue is empty. (Using --log-level=debug will illuminate what
is happening.)
DISTRIBUTED NATURE OF FILESPOOLER
As mentioned, Filespooler is designed to be used as a distributed, asynchronous, ordered
command queue. The homepage contains many more examples. Here is one simple example of
using ssh as a transport to get commands to a remote queue:
tar -cpf - /usr/local | fspl prepare -s ~/state -i - | ssh remote queue-write -q ~/queue
INSTALLATION
fspl is a Rust program. If you don't already have Rust installed, it can be easily
installed from <https://www.rust-lang.org/>.
Once Rust is installed, Filespooler can be installed with this command:
cargo install filespooler
From a checked-out source tree, it can be built by running cargo build --release. The
executable will then be placed in target/release/xbnet.
You can also obtain pre-built binaries for x86_64 Linux from
<https://salsa.debian.org/jgoerzen/filespooler/-/releases> .
ENVIRONMENT
fspl prepare will save certain environment variables to the packet, which will be set
later at process time. fspl {queue,stdin}-process will set a number of useful environment
variables in the execution environment. fspl {queue,stdin}-info will show the environment
that will be passed to the commands. See each of these for further discussion.
EXIT CODE
In general, the commands exit with 0 on success and nonzero on failure. The concept of
success and failure can be complicated in some situations; see the discussion of the
process command.
These situations explicitly cause a nonzero (error) exit code:
• Failure to obtain a lock (see "locking and concurrency" below), but only if a lock is
required; for many commands, no lock is needed.
• An I/O error
• For commands that require a specific job ID (eg, fspl queue-info), no job with that ID
can be located
• While processing, the executed command returns a nonzero exit status and --on-error is
set to Retry (the default)
• In some cases, the presence of multiple files in the queuedir with the same sequence
number. The presence of this condition with commands that take a -j ID option, or with
queue-process in its standard configuration, will cause an error.
• However, this condition is acceptable for queue-ls and queue-process --order-
by=Timestamp.
These situations explicitly terminate with success (0):
• While processing, the --maxjobs limit is reached before some other error causes an
abnormal exit
• An error while running a command while --on-error is set to Delete or Leave
• Files are encountered in the queuedir/jobs directory with unparsable headers. fspl
detects and logs (subject to --log-level) this condition, but does not consider it an
error, on the grounds that the presence of extra data should not prevent the proper
functioning of the queue. This may manifest itself in the queue appearing to have
nothing to do, queue-ls showing fewer jobs than there are files, etc. A common cause of
this may be an incorrect --decoder.
• Zero jobs in the queue, or zero jobs available to process.
LOCKING AND CONCURRENCY
Next to every seqfile on both the sender and within the queue on the recipient is a file
named seqfile.lock. An exclusive lock is held on this file during the following
conditions:
• On the sender with fspl prepare and related functions, briefly while obtaining the next
sequence number. Once this is done, the lock is released, even if the process of
consuming stdin takes a long time.
• On the recipient, when processing the queue with fspl queue-process or other commands
that access the seqfile (eg, fspl queue-set-next).
fspl will exit with an error code if it cannot obtain the lock when it needs it.
These are situations that explicitly do NOT obtain a lock:
• fspl queue-write or other non-fspl method of injecting packets into the queue
• The fspl stdin- series of commands
• Commands that scan the queue without accessing the state of the seqfile. Examples
include queue-ls, queue-info, and queue-payload.
Note that if the queue is being actively processed while a queue-ls is in process, a race
condition is possible if a file disappears between the readdir() call and the time the
file is opened for reading, which could potentially cause queue-ls to fail. queue-ls
intentionally does not attempt to acquire the lock, however, because it would always fail
while the queue is being processed in that case, preventing one from being able to list
the queue at all while long-running jobs are in process.
Note that fspl queue-write does not need to obtain a lock. The fspl stdin- series of
commands also do not obtain a lock.
Taken together, this means that any given queue is intended to be processed sequentially,
not in parallel. However, if parallel processing is desired, it is trivial to iterate
over the jobs and use fspl stdin-process in whatever custom manner you would like. Also,
since queues are so lightweight, there is no problem with creating thousands of them.
INVOCATION: GLOBAL OPTIONS
These options may be specified for any command, and must be given before the command on
the command line.
-l, --log-level LEVEL
Information about the progress of fspl is written to stderr. This parameter
controls how much information is written. In order from most to least information,
the options are: trace, debug, info, warn, error. The default is info.
-V, --version
Print version information and exit
-h, --help
Print help information and exit. Can also be given after a subcommand, in which
case it displays more detailed information about that subcommand.
COMMAND
The subcommand which will be executed. Required unless using --version or --help.
INVOCATION: SUBCOMMANDS
Every subcomand accepts --help to display a brief summary of options, invoked as: fspl
SUBCOMMAND --help .
fspl ... prepare
Generates a packet (job file data) and writes it to stdout. This file can be piped to
other programs (particularly fspl queue-write) or saved directly to disk.
Usage:
fspl prepare [ OPTIONS ] -s FILE [ -- PARAMS... ]
-s, --seqfile FILE
Path to the local seqfile. If it does not already exist, it will be created. If
set to "-", then no sequence file is used and the sequence emitted will always be
1.
-i, --input INPUT
By default, prepare will not read anything as payload. If INPUT is set to "-",
then prepare will read standard input (stdin) and use it as input. Otherwise, if
INPUT is anything other than "-", it is assumed to be a filename, which is opened
and read for input.
-- PARAMS...
If a "--" is present on the command line, everything after it is taken as
parameters to be added to the generated job packet. When the packet is later
processed, if --allow-job-params is given to queue-process or stdin-process, then
these parameters will be appended to the command line of the executed command.
In addition to these options, any environment variable beginning with FSPL_SET_ will be
saved in the packet and will be set in the execution environment at processing time.
fspl ... prepare-fail, prepare-nop
These commands create a non-command packet, one which is either considered to always fail
or to always succeed (nop). These two commands take only one option, which is required:
-s, --seqfile FILE
Path to the local seqfile. Required. If set to "-", then no sequence file is used
and the sequence emitted will always be 1.
fspl ... prepare-get-next
Prints the sequence number that will be used by the next prepare command.
Usage:
fspl prepare-get-next -s FILE
-s, --seqfile FILE
Path to the local seqfile. Required. If set to "-", then no sequence file is used
and the sequence emitted will always be 1.
fspl ... prepare-set-next
Changes the sequence number that will be used by the next prepare command.
Usage:
fspl prepare-set-next -s FILE ID
-s, --seqfile FILE
Path to the local seqfile. Required. ID The numeric ID to set the seqfile to.
fspl ... stdin-info, queue-info
These two commands display information about a given packet. This information is printed
to stdout in a style that is similar to how the shell sets environment variables. In
fact, it shows precisely the environment variables that will be set by a corresponding
process command.
stdin-info expects the packet to be piped in to stdin; queue-info will find it in the
given queue.
This command will not attempt to read the payload of the file; it will only read the
header. (Note that this is not a guarantee that some layer of the system may not try to
read a few KB past the header, merely a note that running this command will not try to
read all of a 1TB packet.)
Usage:
fspl queue-info [ OPTIONS ] -q DIR -j ID
fspl stdin-info
Options (valid for queue-info only):
-q, --queuedir DIR
Path to the local queue directory. Required.
-j, --job ID
Numeric job ID to process. See fspl queue-ls to determine this. Required.
-d, --decoder DECODECMD
Decoder command to run. This string is passed to $SHELL -c. See the above
conversation about decoders. Optional.
Example:
fspl queue-info -q /tmp/queue -j 45 -d zcat
FSPL_SEQ=45
FSPL_CTIME_SECS=1651970311
FSPL_CTIME_NANOS=425412511
FSPL_CTIME_RFC3339_UTC=2022-05-08T00:38:31Z
FSPL_CTIME_RFC3339_LOCAL=2022-05-07T19:38:31-05:00
FSPL_JOB_FILENAME=fspl-29342606-02a0-438c-81f2-efdfb80afbe9.fspl
FSPL_JOB_QUEUEDIR=/tmp/bar
FSPL_JOB_FULLPATH=/tmp/bar/jobs/fspl-29342606-02a0-438c-81f2-efdfb80afbe9.fspl
FSPL_PARAM_1=hithere
FSPL_SET_FOO=bar
Some notes on these variables:
• The FSPL_JOB_FILENAME is relative to the jobs subdirectory of the queue directory.
• The FSPL_JOB_FULLPATH is relative to the current working directory; that is, it is what
was given by -q plus the path within that directory to the filename. It is not
guaranteed to be absolute.
• FSPL_PARAM_n will be set to the optional parameters passed to fspl prepare, with n
starting at 1.
• FSPL_SET_x will reflect any FSPL_SET_x parameters that were in the environment when fspl
prepare was run.
• Filespooler does not enforce limits to environment variable content. If you want to do
something like embed newlines in variable content, Filespooler will happily accept this
(since it is valid POSIX) and handle it properly - but your shell scripts may not be so
lucky. It is advisable that you avoid this and other weird constructions for your
sanity in working with things outside Filespooler - though Filespooler won't prevent you
from doing it.
fspl ... stdin-payload, queue-payload
These two commands extract the payload (if any) from the given packet. This is written to
stdout. No header or other information is written to stdout.
stdin-payload expect the packet to be piped in to stdin; queue-stdout will find it in the
given queue.
The payload will be piped to the command started by the process commands. The payload
will be 0-bytes if -i was not passed to fspl prepare, or if an empty payload was given to
it.
Usage:
fspl queue-payload [ OPTIONS ] -q DIR -j ID
fspl stdin-payload
Options (valid for queue-payload only):
-q, --queuedir DIR
Path to the local queue directory. Required.
-j, --job ID
Numeric job ID to process. See fspl queue-ls to determine this. Required.
-d, --decoder DECODECMD
Decoder command to run. This string is passed to $SHELL -c. See the above
conversation about decoders. Optional.
fspl ... stdin-process, queue-process
Process packet(s). stdin-process will process exactly one packet on stdin. queue-process
will process zero or more packets, depending on the content of the queue and options
given.
Usage:
fspl queue-process [ OPTIONS ] -q DIR COMMAND [ -- PARAMS... ]
fspl stdin-process [ OPTIONS ] COMMAND [ -- PARAMS... ]
Common options:
--allow-job-params
Specifies that optional parameters given to fspl prepare will be passed on the
command line to this command
--ignore-payload
Ignores the payload; does not pipe it to the command.
--timeout SECONDS
Specifies a timeout, in seconds, for the command. If the command has not exited
within that timeframe, SIGKILL is sent to the process. Failing to exit within the
timeout is considered an error for Filespooler's purposes.
COMMAND
The command to run. This is not passed to the shell, so it must point to an
executable. This command will not be run for NOP or Fail packets.
-- PARAMS...
If a "--" is present on the command line, everything after it is taken as
parameters to be sent to the given command. If --allow-job-params is given, then
those parameters will be sent after these.
Options valid only for queue-process:
-q, --queuedir DIR
Path to the local queue directory. Required.
-d, --decoder DECODECMD
Decoder command to run. This string is passed to $SHELL -c. See the above
conversation about decoders. Optional.
-n, --maxjobs JOBS
The maximum number of jobs to process. There is no limit by default.
--never-delete
Never delete the job file after processing in any circumstance, regardless of other
options.
--order-by ORDERING
In what order to process the queue. When Sequence, which is the default, process
the queue in order of sequence number. When set to Timestamp, process the queue in
order of the creation timestamp as it appears in the job header. Note that when
set to Timestamp, the seqfile within the queue is neither used nor changed.
Timestamp implies that you do not care about a strict sequential ordering of items
in cases where items arrive out of order.
--on-error ONERROR
What to do when the supplied command fails (is a fail packet or a command exits
with a nonzero status). If set to Retry, abort processing with a nonzero error
code and leave the packet in the queue to be tried again by a later invocation of
queue-process. If set to Delete, delete the packet from the queue (unless --never-
delete is given), increment the next job counter, and continue processing the queue
normally. If set to Leave, then leave the packet on disk, increment the next job
counter, and continue processing the rest of the queue normally. Retry is the only
option that will cause a failure to not increment the next job counter. Retry is
the default.
--output-to DEST
What to do with the stdout and stderr of the invoked command. If set to PassBoth,
then they are simply written to the stdout/stderr of fspl queue-process. If set to
SaveBoth, then both are added to a file in the queue's jobs directory named
filename.out. This file is up to you to process whenever you wish. The default is
PassBoth.
The environment is set as described above. Note that since no queue directory or filename
is relevant with the stdin-process flavor, those variables are unset under stdin-process.
To skip a failing job at the head of the queue, you can use fspl queue-set-next, or
alternatively, fspl queue-process --on-error Delete --maxjobs 1 to cause it to be deleted.
You would probably not wish to combine this with timestamp ordering.
fspl ... queue-set-next
Changes the sequence number that will be used by the next fspl queue-process command.
Usage:
fspl queue-set-next -q DIR ID
-q, --queuedir DIR
Path to the local queue directory. Required.
--append-only
Creates an append-only queue. ID The numeric ID to set the seqfile to.
fspl ... queue-write
Receives a packet on stdin and writes it to the queue. This command does not bother to
decode, process, or validate the packet in any way. It simply writes it to the queue
safely, using a temporary filename until completely written, at which point it is renamed
to a **fspl-*.fspl** file with a random middle part.
Usage:
fspl queue-write -q DIR
-q, --queuedir DIR
Path to the local queue directory. Required.
fspl ... queue-init
Creates the queue directory and the needed files and subdirectories within it.
Usage:
fspl queue-init -q DIR
-q, --queuedir DIR
Path to the local queue directory. Required.
fspl ... gen-filename
Generates a filename matching the fspl-*.fspl pattern, which will be valid for a job file
in a Filespooler queue. This is often useful when generating a filename that will be used
by a tool other than fspl queue-write.
Usage:
fspl gen-filename
Example:
fspl gen-filename
fspl-b3bd6e63-f62c-49ee-8c46-6677069d2c58.fspl
fspl ... gen-uuid
Generates a random UUID and prints it to stdout. This is generated using the same
algorithm as fspl queue-write uses. It can be used in scripts for making your own unique
filenames.
Usage:
fspl gen-uuid
Example:
fspl gen-uuid
2896c849-37c5-4a6d-8b90-0cf63e3e9daa
fspl show-license
Displays the copyright and license information for fspl.
AUTHOR
John Goerzen <jgoerzen@complete.org>
HOMEPAGE
<https://www.complete.org/filespooler/>
COPYRIGHT AND LICENSE
Copyright (C) 2022 John Goerzen <jgoerzen@complete.org>
This program is free software: you can redistribute it and/or modify it under the terms of
the GNU General Public License as published by the Free Software Foundation, either
version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program.
If not, see <http://www.gnu.org/licenses/>.
AUTHORS
John Goerzen.