Ubuntu Manpage: safecat - safely write data to a file

NAME

       safecat - safely write data to a file

SYNOPSIS

       safecat tempdir destdir

INTRODUCTION

       safecat  is  a  program which implements Professor Daniel Bernstein's maildir algorithm to
       copy stdin safely to a file in a specified directory.  With safecat, the user  is  offered
       two  assurances.   First,  if  safecat  returns a successful exit status, then all data is
       guaranteed to be saved in the destination directory.  Second, if  a  file  exists  in  the
       destination  directory,  placed  there  by  safecat,  then  the  file  is guaranteed to be
       complete.

       When saving data with safecat, the user specifies a destination directory, but not a  file
       name.   The  file name is selected by safecat to ensure that no filename collisions occur,
       even if many safecat processes and other programs implementing the maildir  algorithm  are
       writing  to  the  directory simultaneously.  If particular filenames are desired, then the
       user should rename the file after safecat completes.  In general, when spooling data  with
       safecat,  a  single, separate process should handle naming, collecting, and deleting these
       files.  Examples of such a process are daemons, cron jobs, and mail readers.

RELIABILITY ISSUES

       A machine may crash while data is being written to disk.   For  many  programs,  including
       many  mail  delivery  agents,  this means that the data will be silently truncated.  Using
       Professor Bernstein's maildir algorithm, every file is guaranteed complete or nonexistent.

       Many people or programs may write data to a common "spool" directory.   Systems  like  mh-
       mail  store  files  using  numeric  names in a directory.  Incautious writing to files can
       result in a collision, in which one write succeeds and the other appears  to  succeed  but
       fails.   Common strategies to resolve this problem involve creation of lock files or other
       synchronizing mechanisms, but such mechanisms are subject  to  failure.   Anyone  who  has
       deleted  $HOME/.netscape/lock  in order to start netscape can attest to this.  The maildir
       algorithm is immune to this problem because it uses no locks at all.

THE MAILDIR ALGORITHM

       As described in maildir(5), safecat applies the maildir algorithm by writing data  in  six
       steps.   First,  it stat()s the two directories tempdir and destdir, and exits unless both
       directories exist and are writable.  Second, it stat()s  the  name  tempdir/time.pid.host,
       where  time is the number of seconds since the beginning of 1970 GMT, pid is the program's
       process ID, and host is the host name.  Third, if  stat()  returned  anything  other  than
       ENOENT,  the  program  sleeps for two seconds, updates time, and tries the stat() again, a
       limited number of times.  Fourth, the program creates tempdir/time.pid.host.   Fifth,  the
       program  NFS-writes  the  message  to  the  file.   Sixth, the program link()s the file to
       destdir/time.pid.host.  At that instant the data has been successfully written.

       In addition, safecat starts a 24-hour timer  before  creating  tempdir/time.pid.host,  and
       aborts the write if the timer expires.  Upon error, timeout, or normal completion, safecat
       attempts to unlink() tempdir/time.pid.host.

EXIT STATUS

       An exit status of 0 (success) implies that all data has been safely committed to disk.   A
       non-zero  exit  status  should  be  considered to mean failure, though there is an outside
       chance that safecat wrote the data successfully, but didn't think so.

       Note again that if a file appears in the destination directory, then it is  guaranteed  to
       be complete.

       If  safecat  completes successfully, then it will print the name of the newly created file
       (without its path) to standard output.

SUGGESTED APPLICATIONS

Exciting uses for safecat abound, obviously, but a word may be in order to suggest what
they are.

If you run Linux and use qmail instead of sendmail, you should consider converting your
inbox to maildir for its superior reliability. If your home directory is NFS mounted,
qmail forces you to use maildir.

If you write CGI applications to collect data over the World Wide Web, you might find
safecat useful. Web applications suffer from two major problems. Their performance
suffers from every stoppage or bottleneck in the internet; they cannot afford to introduce
performance problems of their own. Additionally, web applications should NEVER leave the
server and database in an inconsistent state. This is likely, however, if CGI scripts
directly frob some database--particularly if the database is overloaded or slow. What
happens when users get bored and click "Stop" or "Back"? Maybe the database activity
completes. Maybe the CGI script is killed, leaving the DB in an inconsistent state.

Consider the following strategy. Make your CGI script dump its request to a spool
directory using safecat. Immediately return a receipt to the browser. Now the browser
has a complete guarantee that their submission is received, and the perceived performance
of your web application is optimal.

Meanwhile, a spooler daemon notices the fresh request, snatches it and updates the
database. Browsers can be informed that their request will be fulfilled in X minutes.
The result is optimal performance despite a capricious internet. In addition, users can
be offered nearly 100% reliability.

EXAMPLES

       To convince sendmail to use maildir for message delivery, add the following line  to  your
       .forward file:

       |SAFECAT HOME/Maildir/tmp HOME/Maildir/new || exit 75 #USERNAME

       where SAFECAT is the complete path of the safecat program, HOME is the complete path to
       your home directory, and USERNAME is your login name. Making this change is likely to pay
       off; many campuses and companies mount user home directories with NFS.  Using maildir to
       deliver to your inbox folder helps ensure that your mail will not be lost due to some NFS
       error.  Of course, if you are a System Administrator, you should consider switching to
       qmail.

       To run a program and catch its output safely into some directory, you can use a shell
       script like the following.

       #!/bin/bash

       MYPROGRAM=cat              # The program you want to run
       TEMPDIR=/tmp               # The name of a temporary directory
       DESTDIR=$HOME/work/data    # The directory for storing information

       try() { $* 2>/dev/null || echo NO 1>&2 }

       set `( try $MYPROGRAM | try safecat $TEMPDIR $DESTDIR ) 2>&1`
       test "$?" = "0"  || exit -1
       test "$1" = "NO" && { rm -f $DESTDIR/$2; exit -1; }

       This script illustrates the pitfalls of writing secure programs with the shell.  The
       script assumes that your program might generate some output, but then fail to complete.
       There is no way for safecat to know whether your program completed successfully or not,
       because of the semantics of the shell.  As a result, safecat might create a file in the
       data directory which is "complete" but not useful.  The shell script deletes the file in
       that case.

       More generally, the safest way to use safecat is from within a C program which invokes
       safecat with fork() and execve().  The parent process can the simply kill() the safecat
       process if any problems develop, and optionally can try again.  Whether to go to this
       trouble depends upon how serious you are about protecting your data.  Either way, safecat
       will not be the weak link in your data flow.

BUGS

In order to perform the last step and link() the temporary file into the destination
directory, both directories must reside in the same file system. If they do not, safecat
will quietly fail every time. In Professor Bernstein's implementation of maildir, the
temporary and destination directories are required to belong to the same parent directory,
which essentially avoids this problem. We relax this requirement to provide some
flexibility, at the cost of some risk. Caveat emptor.

Although safecat cleans up after itself, it may sometimes fail to delete the temporary
file located in tempdir. Since safecat times out after 24 hours, you may freely delete
any temporary files older than 36 hours. Files newer than 36 hours should be left alone.
A system of data flow involving safecat should include a cron job to clean up temporary
files, or should obligate consumers of the data to do the cleanup, or both. In the case
of qmail, mail readers using maildir are expected to scan and clean up the temporary
directory.

The guarantee of safe delivery of data is only "as certain as UNIX will allow." In
particular, a disk hardware failure could result in safecat concluding that the data was
safe, when it was not. Similarly, a successful exit status from safecat is of no value if
the computer, its disks and backups all explode at some subsequent time.

In other words, if your data is vital to you, then you won't just use safecat. You'll
also invest in good equipment (possibly including a RAID disk), a UPS for the server and
drives, a regular backup schedule, and competent system administration. For many
purposes, however, safecat can be considered 100% reliable.

Also note that safecat was designed for spooling email messages; it is not the right tool
for spooling large files--files larger than 2GB, for example. Some operating systems have
a bug which causes safecat to fail silently when spooling files larger than 2GB. When
building safecat, you can take advantage of conditional support for large files on Linux;
see conf-cc for further information.

CREDITS

       The maildir algorithm was devised by Professor Daniel Bernstein, the author of qmail.
       Parts of this manpage borrow directly from maildir(5) by Professor Bernstein.  In
       particular, the section "THE MAILDIR ALGORITHM" transplants his explanation of the maildir
       algorithm in order to illustrate that safecat complies with it.

       The original code for safecat was written by the present author, but was since augmented
       with heavy borrowings from qmail code.  However, under no circumstances should the author
       of qmail be contacted concerning safecat bugs; all are the fault, and the responsibility,
       of the present author.

       Copyright (c) 2000, Len Budney. All rights reserved.