Provided by: cnews_cr.g7-39_i386 bug

NAME

       sendbatches, batchsplit - news batching to other sites
       batcher, batchih, batchsm, batchra - news-batch preparation
       compcun - news-batch compression
       c7encode, bencode - compressed-news-batch encoding
       viauux - news-batch transmission via uucp
       viamail - news-batch transmission via mail
       viainews, viarsh - news-batch transmission by misc. means

SYNOPSIS

       /usr/lib/news/batch/sendbatches [ -p ] [ -c class ] [ site ... ]

       .../batchsplit batchsize maxsize count
       .../batcher [ listfile ]
       .../batchih [ listfile ]
       .../batchsm [ listfile ]
       .../batchra [ listfile ]
       .../compcun [ compressoptions ]
       .../c7encode
       .../bencode [ file ]
       .../viauux  [ -ggrade ] [ -z ] [ -n ] [ -d decompressor ] [ -c ] [ site
       ]
       .../viamail [ -e ] [ 0-@ ] [ site ]
       .../viainews
       .../viarsh [ site ]

DESCRIPTION

       Sendbatches administers batched transmission of news  to  other  sites.
       It  should  be  run periodically, as the owner of the news database, by
       cron(8) or similar means.  It  prepares  and  sends  batches  of  news,
       subject  to  restrictions  on  available  space  and length of outbound
       queues.

       Normally, sendbatches does locking to ensure that only one  sendbatches
       is running at a time.  The -p option suppresses the locking and permits
       parallel sendbatches runs, although  lower-level  locking  is  done  to
       ensure  that  only  one is trying to prepare batches for any particular
       site at a given time.  Parallel  sendbatches  runs  impose  very  heavy
       system  loads  but  may  be  useful  to systems with extensive hardware
       parallelism and many outbound news feeds.

       Each site that can have batches sent to it needs a site directory under
       /var/spool/news/out.going.   If  sendbatches  is  invoked with specific
       sites given, it considers batching  for  those  sites,  only,  in  that
       order.   By  default,  sendbatches  consults  the  batchparms file (see
       below) to determine what to do:  If  there  is  a  /default/  entry  in
       batchparms,  sendbatches will consider batching for all sites that have
       directories in  /var/spool/news/out.going,  in  oldest-first  order  by
       modification  time  of  the directory.  If there is no /default/ entry,
       sendbatches considers batching for those sites named in batchparms,  in
       the order named.

       To use the batcher, names of files to be sent to a specific site should
       be appended to a togo file in its site directory.  The batcher  expects
       the  lines  in togo to have two fields, a filename (as a full pathname,
       or relative to /var/spool/news) of an article and its size in bytes.  A
       missing size field is arbitrarily assumed to be a default average.

       Sendbatches  uses  a  number of auxiliary programs to do the real work.
       The search path it uses to find  them  includes,  in  order,  the  site
       directory   for   the   site   in   question,   /var/lib/news/bin,  and
       /usr/lib/news/batch.   This  permits  per-site  and   per-news-database
       overrides  of  the  default  behaviors.  Sendbatches provides all these
       programs with environment variables NEWSSITE, containing  the  name  of
       the  site  that  batches  are  being  prepared  for,  and  NEWSSITEDIR,
       containing the full pathname of the site directory, in case  these  are
       useful in customization.

       The names of most of the auxiliary programs, and some other parameters,
       are taken from the file /var/lib/news/batchparms, an ASCII  text  file.
       Empty  lines  and  lines  starting  with  ‘#’ are ignored.  Other lines
       specify the behavior for sites, one line per site.  A site line is four
       fields, separated by white space.

       The  first field is the site name.  A line whose site name is /default/
       specifies what parameters should  be  used  for  sites  not  explicitly
       mentioned.  (The presence or absence of such a line also influences the
       behavior of sendbatches when invoked without arguments; see above.)

       The second field is the class of the site.  If sendbatches  is  invoked
       with  the  -c  option,  it  attempts  batching  only  for  sites of the
       specified class.  A class is a single letter,  by  convention  ‘u’  for
       UUCP  feeds  and  ‘n’  for  NNTP  feeds; user-defined classes should be
       uppercase letters.

       If the  character  ‘!’  appears  anywhere  in  the  class  field  of  a
       batchparms line, that disables the line.  Any site whose batching would
       have been controlled by that line will have no batching  attempted  for
       it  under  any  circumstances.   This provides a way to turn a site off
       temporarily.

       The third  field  is  the  size  of  batches  to  be  prepared  (before
       compression),  in  bytes.   It may optionally be two sizes separated by
       ‘-’, in which case the first is the nominal size and the second is  the
       absolute maximum.  If only one size is given, that is the nominal size,
       and the absolute maximum is three times that.

       The fourth field  is  the  maximum  length  of  the  output  queue  for
       transmission  to  that site.  If it is ‘-’, no queue-length limiting is
       done.

       The fifth field, which may contain white space,  is  the  command  line
       (normally  a pipeline of three-four programs, possibly with options) to
       be used to build, compress, and transmit  batches  to  that  site.   It
       receives  the  contents of the togo file on standard input.  It may not
       contain any single quotes (’).

       For  each  site  being  considered  for  batches,   sendbatches   first
       determines  whether  there  are  in  fact  any  articles to be batched.
       Assuming there are, sendbatches then finds the batchparms line for that
       site  and  (if  queue-length  limiting  is  done for that site) invokes
       queuelen (see newsaux(8CN)) to find out the size of the outbound  queue
       for the site.  Sendbatches limits the number of batches prepared to the
       minimum of the limits implied by queue lengths and available space.

       Sendbatches uses batchsplit as necessary to slice  chunks  out  of  the
       togo  file, each chunk containing the togo lines for a batch limited to
       the nominal size.  Exception:  a single article bigger than the nominal
       size  will  still  go out as one batch, provided it does not exceed the
       absolute-maximum size.

       Each chunk is then processed through the command  line,  which  usually
       consists  of  a batch preparer (typically batcher), which assembles the
       articles into a batch, a batch compressor  (typically  compcun),  which
       performs  compression,  possibly  a  batch  encoder,  which encodes the
       compressed batch against  the  vagaries  of  ill-designed  transmission
       channels,  and  a batch transmitter (typically viauux), which sends the
       batch on its way (e.g. enqueues it for transmission).  All are run with
       /var/spool/news  as the current directory, so non-absolute pathnames in
       the chunk are valid filenames; NOTE that this represents a change  from
       earlier releases, in which only the preparer ran there.

       If  there  is  anything left in the file togo.leftover after a chunk is
       processed, sendbatches assumes this is the portion of the  chunk  which
       could not be processed at this time.  If/when this happens, sendbatches
       replaces the chunk file with the contents of togo.leftover and attempts
       no further batching to that site on this run.

       Batch preparers in the standard distribution are:

              batcher  Normal batching.

              batchih  Ihave-sending  part  of  uucp  ihave/sendme  (not to be
                       confused with NNTP).

              batchsm  Sendme-sending part of ihave/sendme.

              batchra  Requested-article-sending part of ihave/sendme.

       Batchih, batchsm, and batchra have to map from the phony ‘‘site  name’’
       given  in  their  batchparms  line  to the name of the site they should
       actually send to; they do this  by  stripping  off  the  last  ‘.’  and
       everything  that  follows  (usually ‘.ihave’ or ‘.sendme’ respectively,
       but on machines which limit the size of filenames these may have to  be
       shortened).

       Caution:  batchih  and batchsm do their work by constructing an article
       and feeding it to inews, which means that the batch size must be within
       what inews will accept, if it imposes limits (see inews(1CN)).

       Batch compressors can include ordinary programs like compress and gzip,
       but one special compressor is provided by the distribution:

              compcun  Compression  with  compress  plus  the  silly   B-news-
                       compatible  ‘#! cunbatch’ header.  Options, if any, are
                       passed to compress.

       Batch encoders supplied with the distribution are:

              bencode  Encodes 8-bit data  using  only  the  ASCII  characters
                       ‘‘A’’ - ‘‘Z’’, ‘‘a’’ - ‘‘z’’, ‘‘0’’ - ‘‘9’’, ‘‘+’’, and
                       ‘‘-’’.  The ASCII characters blank, newline, and  ‘‘/’’
                       also  appear  in the encoded file, but do not represent
                       encoded bits.  The encoded data is  terminated  with  a
                       byte  count  and  cyclic redundancy check for detecting
                       corruption.  This ought to suffice to get data  through
                       almost any network.

              c7encode Encodes  8-bit  data  into  a  7-bit form optimized for
                       transmission by uucp ‘f’  protocol.   The  encoding  is
                       complex and bizarre.  Obsolete; use bencode instead.

              nencode  Feebly  attempts to protect text (not compressed!) data
                       by prepending a blank line to the input and an  ‘N’  to
                       each  line.   Obsolete and not recommended; use bencode
                       instead.

       Most transmitters take an optional site argument, using  the  value  of
       the  NEWSSITE  variable if no argument is supplied.  Batch transmitters
       in the standard distribution are:

              viauux    Normal transmission via UUCP.  -z or -n may be used to
                        feed the corresponding option to uux.  A -g option may
                        be used to feed the grade to uux (the default is grade
                        ‘d’).    A   -d  option  may  be  used  to  specify  a
                        decompressor program  (e.g.,  gunzip)  to  the  site’s
                        rnews; beware that old rnewses may not recognize this.
                        -c may be used to  request  execution  of  the  site’s
                        cunbatch program rather than rnews.

              viamail   Mail  the batch to ‘site!rnews’.  -e specifies mailing
                        to  ‘site!enews’  instead.   -@  specifies  use  of  @
                        syntax, e.g. ‘rnews@site’, instead of ! syntax.

              viainews  Feed  the  batch back to inews (arguments are ignored)
                        (normally useful only for ihave/sendme).

              viarsh    Use rsh  to  run  rnews  on  the  site  via  Ethernet,
                        Internet,  etc.   (the directory containing rnews must
                        be in the default PATH on site).

       Sendbatches  logs  some  information  about  sites  with  backlogs   in
       /var/lib/news/batchlog  (see also newsdaily(8CN)).  This is intended to
       help detection and diagnosis of flow problems.

FILES

       /var/lib/news/LOCKbatch           lock for sendbatches
       /var/lib/news/LOCKexplode         overall batch-file lock (used by batchsplit)
       /var/lib/news/L.*                 lock temporaries
       /var/spool/news/out.going/*       batch directories
       /var/spool/news/out.going/*/LOCK  per-site locks
       /var/spool/news/out.going/*/L.*   per-site lock temporaries
       /var/lib/news/batchparms          parameter file

       also see text

SEE ALSO

       inews(1CN), compress(1), uux(1), relaynews(8CN), rnews(8CN)

DIAGNOSTICS

       Complaints, if any, from sendbatches and its auxiliaries are mailed via
       report.

HISTORY

       Written  at  University  of  Toronto  as part of the C News project.  A
       number of the ideas came from Chris Lewis, who was then with  Motorola.
       Bencode  written  at  University  of  Waterloo  by  Reg Quinton and Ken
       Lalonde.

BUGS

       Sendbatches estimates available space without considering  the  effects
       of compression; this is usually too conservative.

       Viarsh does not incorporate a spooling subsystem, so a slow site stalls
       the entire batching system and a non-responding site loses news.  It is
       not  recommended  for  bulk  transmission  or where high reliability is
       essential.

       There ought to be an encoder using MIME’s base64 encoding and  suitable
       descriptive headers.

       The  logging  is overly simplistic and doesn’t work well when selective
       batching (controlled by site arguments, classes, or disabling) is being
       done.

                                 10 March 1995                  NEWSBATCH(8cn)