Provided by: libparanoid-perl_2.10-1_all 
NAME
Paranoid::IO::FileMultiplexer - File Multiplexer
VERSION
$Id: lib/Paranoid/IO/FileMultiplexer.pm, 2.10 2022/03/08 00:01:04 acorliss Exp $
SYNOPSIS
$obj = Paranoid::IO::FileMultiplexer->new(
file => $fn,
readOnly => 0,
perms => $perms,
blockSize => $bsize,
);
$header = $obj->header;
$rv = $obj->chkConsistency;
$rv = $obj->addStream($name);
$rv = $obj->strmSeek($sname, $pos, $whence);
$rv = $obj->strmTell($sname);
$bw = $obj->strmWrite($sname, $content);
$br = $obj->strmRead($stream, \$content, $bytes);
$bw = $obj->strmAppend($sname, $content);
$bw = $obj->strmTruncate($sname, $neos);
DESCRIPTION
This class produces file multiplexer objects that multiplex I/O streams into a single
file. This allows I/O patterns that would normally be applied to multiple files to be
applied to one, with full support for concurrent access by multiple processes on the same
system.
At its most basic, one could use these objects as an archive format for multiple files.
At its most complex, this could be a database backend file, similar to sqlite or Berkeley
DB.
This does require flock support for the file.
CAVEATS FOR USAGE
This class is built essentially as a block allocation tool, which does have some side
effects that must be anticipated. Full support is available for both 32-bit and 64-bit
file systems, and files produced can be exchange across both types of platforms with no
special handling, at least until the point the file grows beyond the capabilities of a 32
bit platform. Similarly, portability should work fine across both endian platforms.
That said, the simplicity of this design did require some compromises, the first being the
number of supported "streams" that can be stored inside a single file. That is a function
of the block size chosen for the file. All allocated streams are tracked in the file
header block, so the number of streams is constrained by the number that can be recorded
in that block.
Likewise, the maximum size of a stream is also limited by the block size, since the stream
head block can only track so many block allocation tables, and each block allocation table
can only track so many data blocks.
Practically speaking, for many use cases this should not be an issue, but you can get an
idea of the impact on both 32-bit and 64-bit systems like so:
32b/4KB 64b/4KB
--------------------------------------------------------------------------
Max File Size: 4294967295 (4.00GB) 18446744073709551615 (16.00EX)
Max Streams: 135 135
Max Stream Size: 1052872704 (1004.10MB) 1052872704 (1004.10MB)
32b/8KB 64b/8KB
--------------------------------------------------------------------------
Max File Size: 4294967295 (4.00GB) 18446744073709551615 (16.00EX)
Max Streams: 272 272
Max Stream Size: 4294967295 (4.00GB) 8506253312 (7.92GB)
As you can see, 8KB blocks will provide full utilization of your file system capabilities
on a 32-bit platform, but on a 64-bit platform, you are still artificially capped on how
much data can be stored in an individual stream. The number of streams will always
limited identically on both platforms based on the block size.
NOTE: The actual limits of file sizes aren't dependent upon the native size of longs or
quads, but the file system design, itself. Some file systems designed for 32-bit
processors reserved the highest bit, which made the highest addressable space in a file
2GB instead of 4GB. Other filesystems had limits that were a function of inode size and
other aspects of the formatted file system. End sum, the true limit for file size may be
outside of the ability for this module to detect and accomodate gracefully.
One final caveat should be noted regarding I/O performance. The supported block sizes are
intentionally limited in hopes of avoiding double-write penalties due to block alignment
issues on the underlying file system. At the same time, the block size also serves as a
kind of crude tuning capability for the size of I/O operations. No individual I/O,
whether read or write, will exceed the size of a block. You, as the developer, can call
the class API with reads of any size you wish, of course, but behind the scenes it will be
broken up into block-sized reads at most.
For those reasons, when choosing your block size one should choose based on the best
compromise between I/O performance and the minimum number of streams (or maximum stream
size) anticipated.
As a final note, one should also remember that space is allocated to the file in block
sized chunks. That means creating a new file w/1MB block size, containing one stream, but
with nothing written to the stream, will create a file 4MB in size. That's due to the
preallocation of the file header, a stream header, the stream's first block allocation
table, and an initial data block.
SUBROUTINES/METHODS
new
$obj = Paranoid::IO::FileMultiplexer->new(
file => $fn,
readOnly => 0,
perms => $perms,
blockSize => $bsize,
);
This class method creates new objects for accessing the contents of the pass file. It
will create a new file if missing, or open an existing file and retrieve the metadata for
tuning.
Only the file name is mandatory. Block size defaults to 4KB, but if specified, can
support from 4KB to 1MB block sizes, as long as the block size is a multiple of 4KB.
header
$header = $obj->header;
This method returns a reference to the file header block object. Typically, this has no
practical value to the developer, but the file header does provide a model method that
returns a hash with some predicted sizing limitations. if you want to know the maximum
number of supported streams or the maximum size of an individual stream, this could be
useful. Calling any other method for that class, however, could cause corruption of your
file.
chkConsistency
$rv = $obj->chkConsistency;
This method performs a high-level consistency check of the file structure. At this time
it is limited to ensuring that every header block (file, stream, and BAT) has a viable
signature, and all records inside those blocks are allocated and match signatures where
appropriate.
If this method detects any inconsistencies it will mark the object as corrupted, which
will prevent any further writes to the file in hopes that further corruption can be
avoided.
The file format of this multiplexer is such that a good deal of data can be recovered even
with the complete loss of the file header. Corruption in a stream header can even be
recovered from. Only the loss of a BAT header can prevent data from being recovered, but
even then that will only impact the stream it belongs to. It should not impact other
streams.
Take this with a grain of salt, of course. There are always caveats to that rule,
depending on whether the corruption has been detected prior to dangerous writes. Every
read and write to a stream triggers a few basic consistency checks prior to progressing,
but they are not as thorough as this method's process, lest it have and adverse impact on
performance.
This returns a boolean value.
addStream
$rv = $obj->addStream($name);
This method adds a stream to the file, triggering the automatic allocation of three blocks
(a stream header, the first stream BAT, and the first data block). It returns a boolean
value, denoting success or failure.
strmSeek
$rv = $obj->strmSeek($sname, $pos, $whence);
This method acts the same as the core sysseek, taking the same arguments, but with the
substitution of the stream name for the file handle. It's return value is also the same.
Note that the position returned is relative to the data stream, not the file itself.
strmTell
$rv = $obj->strmTell($sname);
This method acts the same as the core tell, taking the same arguments, but with the
substitution of the stream name for the file handle. Like strmSeek, the position returned
is relative to the data stream, not the file itself.
strmWrite
$bw = $obj->strmWrite($sname, $content);
This method acts similarly to a very simplifed syswrite. It does not support length and
offset arguments, only the content itself. It will presume that the stream position has
been adjusted as needed prior to invocation.
This returns the number of bytes written. If everything is working appropriately, that
should match the byte length of the content itself.
strmRead
$br = $obj->strmRead($stream, \$content, $bytes);
This method acts similarly to a very simplified sysread. It does not support offset
arguments, only a scalar reference and the number of bytes to read. It also presumes that
the stream position has been adjusted as needed prior to invocation.
This returns the number of bytes read. Unless you've asked for more data than has been
written to the stream, this should match the number of bytes requested.
strmAppend
$bw = $obj->strmAppend($sname, $content);
This method acts similarly to Paranoid::IO's pappend. It always seeks to the end of the
written data stream before appending the requested content. Like strmWrite, it will
return the number of bytes written. Like pappend, it does not move the stream position,
should you perform additional writes or reads.
strmTruncate
$bw = $obj->strmTruncate($sname, $neos);
This method acts similarly to truncate. It returns a boolean value denoting failure or
success.
DESTROY
Obviously, one would never need to call this directly, but it is documented here to inform
the developer that once an object goes out of scope, it will call pclose on the file,
explicitly closing and purging any cached file handles from Paranoid::IO's internal cache.
DEPENDENCIES
o Carp
o Fcntl
o Paranoid
o Paranoid::Debug
o Paranoid::IO
o Paranoid::IOFileMultiplexer::Block::FileHeader
o Paranoid::IOFileMultiplexer::Block::StreamHeader
o Paranoid::IOFileMultiplexer::Block::BATHeader
BUGS AND LIMITATIONS
AUTHOR
Arthur Corliss (corliss@digitalmages.com)
LICENSE AND COPYRIGHT
This software is free software. Similar to Perl, you can redistribute it and/or modify it
under the terms of either:
a) the GNU General Public License
<https://www.gnu.org/licenses/gpl-1.0.html> as published by the
Free Software Foundation <http://www.fsf.org/>; either version 1
<https://www.gnu.org/licenses/gpl-1.0.html>, or any later version
<https://www.gnu.org/licenses/license-list.html#GNUGPL>, or
b) the Artistic License 2.0
<https://opensource.org/licenses/Artistic-2.0>,
subject to the following additional term: No trademark rights to "Paranoid" have been or
are conveyed under any of the above licenses. However, "Paranoid" may be used fairly to
describe this unmodified software, in good faith, but not as a trademark.
(c) 2005 - 2021, Arthur Corliss (corliss@digitalmages.com) (tm) 2008 - 2021, Paranoid Inc.
(www.paranoid.com)