Ubuntu Manpage: cycbuff.conf - Configuration file for INN CNFS storage method

NAME

       cycbuff.conf - Configuration file for INN CNFS storage method

DESCRIPTION

       This file defines the cyclical buffers that make up the storage pools for CNFS (Cyclic
       News File System).  Some options controlling the behavior of the CNFS storage system can
       also be set here.  cycbuff.conf is required if the CNFS (Cyclic News File System) storage
       method is used.  INN will look for it in pathetc (as set in inn.conf).

       CNFS stores articles in logical objects called metacycbuffs.  Each metacycbuff is in turn
       composed of one or more physical buffers called cycbuffs.  As articles are written to the
       metacycbuff, each article is written to the next cycbuff in the list in a round-robin
       fashion (unless "sequential" mode is specified, in which case each cycbuff is filled
       before moving on to the next).  This is so that you can distribute the individual cycbuffs
       across multiple physical disks and balance the load between them.  Note that in order to
       use any cycbuff larger than 2 GB, you need to build INN with the --enable-largefiles
       option.

       For information about how to configure INN to use CNFS, see storage.conf(5).

       Blank lines and lines beginning with a hash sign ("#") are ignored.  All other lines must
       be of one of the following forms:

           cycbuffupdate:<interval>
           refreshinterval:<interval>
           cycbuff:<name>:<file>:<size>
           metacycbuff:<name>:<buffer>[,<buffer>,...][:<mode>]

       (where items enclosed in [] are optional).  Order is mostly not significant, but all
       cycbuff lines must occur before all metacycbuff lines.  Long lines can be continued on the
       next line by ending the line with a backslash ("\").

       cycbuffupdate:<interval>
           Sets the number of articles written before the cycbuff header is written back to disk
           to <interval>.  Under most operating systems, the header doesn't have to be written to
           disk for the updated data to be available to other processes on the same system that
           are reading articles out of CNFS, but any accesses to the CNFS cycbuffs over NFS will
           only see the data present at the last write of the header.  After a system crash, all
           updates since the last write of the CNFS header may be lost.  The default value, if
           this line is omitted, is 25, meaning that the header is written to disk after every 25
           articles stored in that cycbuff.

       refreshinterval:<interval>
           Sets the interval (in seconds) between re-reads of the cycbuff header to <interval>.
           This primarily affects nnrpd and controls the frequency with which it updates its
           knowledge of the current contents of the CNFS cycbuffs.  The default value, if this
           line is omitted, is 30.

       cycbuff:<name>:<file>:<size>
           Configures a particular CNFS cycbuff.  <name> is a symbolic name for the buffer, to be
           used later in a metacycbuff line.  It must be no longer than seven characters.  <file>
           is the full path to the buffer file or block device, and must be no longer than 63
           characters.  <size> is the length of the buffer in kilobytes (1 KB is 1024 bytes).  If
           <file> is not a block device, it should be <size> * 1024 bytes long.

           If you're trying to stay under 2 GB, keep your sizes below 2097152.

       metacycbuff:<name>:<buffer>[,<buffer>,...][:<mode>]
           Specifies a collection of CNFS buffers that make up a single logical storage location
           from the perspective of INN.  Metacycbuffs are referred to in storage.conf as storage
           locations for articles, so in order to actually put articles in a cycbuff, it has to
           be listed as part of some metacycbuff which is then referenced in storage.conf.

           <name> is the symbolic name of the metacycbuff, referred to in the options: field of
           "cnfs" entries in storage.conf.  It must be no longer than eight characters.  <buffer>
           is the name of a cycbuff (the <name> part of a cycbuff line), and any number of
           cycbuffs may be specified, separated by commas.

           If there is more than one cycbuff in a metacycbuff, there are two ways that INN can
           distribute articles between the cycbuffs.  The default mode, "INTERLEAVE", stores the
           articles in each cycbuff in a round-robin fashion, one article per cycbuff in the
           order listed.  If the cycbuffs are of wildly different sizes, this can cause some of
           them to roll over much faster than others, and it may not give the best performance
           depending on your disk layout.  The other storage mode, "SEQUENTIAL", instead writes
           to each cycbuff in turn until that cycbuff is full and then moves on to the next one,
           returning to the first and starting a new cycle when the last one is full.  To specify
           a mode rather than leaving it at the default, add a colon and the mode ("INTERLEAVE"
           or "SEQUENTIAL") at the end of the metacycbuff line.

       innd only reads cycbuff.conf on startup, so if you change anything in this file and want
       innd to pick up the changes, you have to use "ctlinnd xexec innd"; "ctlinnd reload all ''"
       is not sufficient.

       When articles are stored, the cycbuff into which they're stored is saved as part of the
       article token.  In order for INN to retrieve articles from a cycbuff, that cycbuff must be
       listed in cycbuff.conf.  However, if INN should not write to a cycbuff, it doesn't need to
       be (and shouldn't be) listed in a metacycbuff.

       This provides an easy way to retire a cycbuff.  Just remove it from its metacycbuff,
       leaving in the cycbuff line, and restart innd (with, for example, "ctlinnd xexec innd").
       No new articles will be put into the cycbuff, but neither will any articles expire from
       it.  After you no longer need the articles in the cycbuff, just remove it entirely from
       cycbuff.conf.  Then all of the articles will appear to have been deleted to INN, and the
       next nightly expire run will clean up any remaining references to them.

       Adding a new cycbuff just requires creating it (see below), adding a cycbuff line, adding
       it to a metacycbuff, and then restarting innd.

CREATING CYCBUFFS

When creating a new cycbuff, there are two different methods for creating the buffers in
which the articles will be stored.

1. Create a large file on top of a regular file system. The easiest way to do this is
probably with dd(1), using a command like:

dd if=/dev/zero of=/path/to/cycbuff bs=1024 count=<size>

where <size> is the size from the cycbuff line in cycbuff.conf. INSTALL contains a
script that will generate these commands for you from your cycbuff.conf file.

This is the simplest method, but has the disadvantage that very large files on regular
file systems can be fairly slow to access, particularly at the end of the file, and
INN incurs unnecessary file system overhead when accessing the cycbuff.

2. Use block devices directly. If your operating system allows you to call mmap() on
block devices (Solaris and recent versions of Linux do, FreeBSD at last report does
not), this is the recommended method since you can avoid all of the native file system
overhead.

Note that some OSes do not support files larger than 2 GB, which will limit the size
you can make a single cycbuff, but you can still combine many cycbuffs into each
metacycbuff. Very old versions of Linux (before 2.4 kernels, that raised the limit to
2 TB) are known to have this limitation; FreeBSD does not. Some OSes that support
large files don't support direct access to block devices for large partitions (Solaris
prior to Solaris 7, or not running in 64-bit mode, is in this category); on those
OSes, if you want cycbuffs over 2 GB, you'll have to use regular files. If in doubt,
keep your cycbuffs smaller than 2 GB.

Partition the disk to make each partition equal to or smaller than 2 GB. If you're
using Solaris, set up your partitions to avoid the first cylinder of the disk (or
otherwise the cycbuff header will overwrite the disk partition table and render the
cycbuffs inaccessible). Then, create device files for each block device you're going
to use.

It's not recommended to use the block device files in /dev, since the news system
doesn't have permission to write to them and changing the permissions of the system
device files may affect something else. Instead, use mknod(1) to create a new set of
block devices (in somewhere like pathspool/cycbuffs that's only writable by the news
user). To do this, run "ls -Ll" on the devices in /dev that correspond to the block
devices that you want to use. The major and minor device numbers are in the fifth and
sixth columns (right before the date), respectively. Then run mknod like:

mknod <file> b <major> <minor>

where <file> is the path to the device to create (matching the <file> part of the
cycbuff line) and <major> and <minor> are the major and minor device numbers as
discovered above.

Here's a short script to do this when given the path to the system device file as an
argument:

#!/bin/sh
base=`echo "$1" | sed 's%.*/%%'`
major=`ls -Ll "$1" | awk '{print $5}' | tr -d ,`
minor=`ls -Ll "$1" | awk '{print $6}`
mkdir -p <pathspool in inn.conf>/cycbuffs
mknod <pathspool>/cycbuffs/"$base" b "$major" "$minor"
chown news:news <pathspool>/cycbuffs/"$base"
chmod 644 <pathspool>/cycbuffs/"$base"

Make sure that the created files are owned by the news user and news group, as
specified at configure time (the default being "news" for both). Also make sure that
the permissions on the devices allow the news user to read and write, and if you want
other users on the system to be able to use sm to retrieve articles, make sure they're
world-readable.

Once you have everything configured properly and you start innd, you should see messages
in news.notice that look like:

innd: CNFS-sm No magic cookie found for cycbuff ONE, initializing

where "ONE" will be whatever you called your cycbuff.

HISTORY

       Written by Katsuhiro Kondou <kondou@nec.co.jp> for InterNetNews.  Rewritten into POD by
       Russ Allbery <rra@stanford.edu>.

       $Id: cycbuff.conf.pod 8987 2010-02-16 19:48:45Z iulius $

NAME

DESCRIPTION

CREATING CYCBUFFS

HISTORY

SEE ALSO