Provided by: libparanoid-perl_2.10-1_all 
NAME
Paranoid::IO - Paranoid IO support
VERSION
$Id: lib/Paranoid/IO.pm, 2.10 2022/03/08 00:01:04 acorliss Exp $
SYNOPSIS
use Fcntl qw(:DEFAULT :flock :mode :seek);
use Paranoid::IO qw(:all);
# Implicit open
$chars = pread("./foo.log", $in, 2048);
# Implcit write/append
$chars = pwrite("./bar.log", $out);
$chars = pappend("./bar.log", $out);
# Adjust block read size
PIOBLKSIZE = 8192;
# Adjust max file size for file scans
PIOMAXFSIZE = 65536;
# Enable flock lock stack
PIOLOCKSTACK = 1;
# Explicit open with explicit locking
$fh = popen($filename, O_RDWR | O_CREAT | O_TRUNC, 0600);
$rv = pseek($filename, 0, SEEK_END);
$rv = pflock($filename, LOCK_EX);
if ($rv > 0) {
pseek($filename, 0, SEEK_SET) && ptruncate($filename);
}
$rv = pwrite($filename, $text)
$rv = ptell($filename);
$rv = plockstat($filename);
# Calls that ignore file locks
$rv = pnlwrite($filename, $text)
$rv = pnlappend($filename, $text)
$rv = pnlread($filename, $text)
# After fork
$rv = preopen();
$rv = pclose($filename);
$rv = pcloseAll();
DESCRIPTION
Paranoid::IO is intended to make basic file I/O access easier while keeping with the
tenets of paranoid programming. Most of these calls are essentially wrappers for the
basic system calls (exposed as sysopen, syswrite, etc.) with some additional logic to
reduce the need to explicitly code every step of normal safe access rules, such as file
locking. In the most basic of usage patterns, even explicitly opening files isn't
necessary.
For the most part the system calls that are wrapped here act identically as the underlying
calls, both in the arguments they take and the values they return. The one notable
difference, however, is the popen function itself. A glob variable isn't passed for
assignation since this module stores those references internally along with some meta
data, so popen returns file handles directly.
That semantic, however, is what gives the rest of the functions the flexibility of
accepting either a file name or a file handle to work on. In the case of file names some
of these functions can open files automatically, and the rest of the features are granted
automatically.
In the case of passing file handles the full feature set of this module is only available
if the file handle was originally opened with popen. The calls will still work even if it
wasn't, but some of the safety features, like being fork-safe, won't have the meta data to
work properly.
The features provided by this module are:
• Opportunistic file access
• File handle caching
• Fork-safe file access
• Inherent file locking
• O_APPEND access patterns where needed even for files not opened with O_APPEND
• Intelligent file tracking
• Optional flock lock stack for transactional I/O patterns
The following sections will explain each feature in more depth:
Opportunistic file access
Opportunistic file access is defined here as not needing the explicit I/O handling for
what can be implied. For instance, to read content from a file one can simply use the
pread function without having to open and apply a shared file lock. In a similar manner
one should be able to write or append to a file. Files are automatically opened (with the
file mode being intuited by the type of call) as needed. Only where more complicated
access patterns (such as read/write file handles) should an explicit popen call be needed.
Opportunism is limited to where it makes sense, however. Files are not opportunistically
opened if the first I/O call is to pseek or ptell. The intent of the file I/O (in regards
to read/write file modes) is impossible to tell within those calls.
File handle caching
This module provides a replacement for Perl's internal sysopen, which should be used even
where read/write file access is necessary. One key benefit for doing so is that it
provides internal file handle caching based on the file name. All the additional
functions provided by this module use it internally to retrieve that cached file handle to
avoid the overhead of repetitive opening and closing of files.
Fork-safe file access
A greater benefit of popen, however, is in it's fork-safe behavior. Every call checks to
see if the file handle it has was inherited from its parent, and if so, transparently
closed and reopened so I/O can continue without both processes conflicting over cursor
positions and buffers. After files are reopened read cursors are placed at the same
location they were prior to the first I/O access in the child.
File modes are preserved without the obvious conflicts of intent as well. Files opened in
the parent with O_TRUNC are reopened without that flag to prevent content from being
clobbered.
Inherent file locking
Except where explicitly ignored (like for pnlread) all read, write, and append operations
use locking internally, alleviating the need for the developer to do so explicitly. Locks
are applied and removed as quickly as possible to facilitate concurrent access.
If you're managing flocks directly, however, all of the read/write functions in this
module not only support an option boolean argument to disable internal flocking, but also
have pnl* wrapper functions that set that argument for you.
O_APPEND access patterns
pappend allows you to mimic O_APPEND access patterns even for files that weren't
explicitly opened with O_APPEND. If you open a file with O_RDWR you can still call
pappend and the content will be appended to the end of the file, without moving the file's
cursor position for regular reads and writes.
Intelligent file tracking
popen caches file handles by file name. If files are opened with relative paths this has
the potential to cause some confusion if the process or children are changing their
working directories. In anticipation of this popen also tracks the real path (as resolved
by the realpath system call) and file name. This way you can still access the same file
regardless of the process or its children's movements on the file system.
This could be, however, a double-edged sword if your program intends to open identically
named files in multiple locations. If that is your intent you would be cautioned to avoid
using relative paths with popen.
Optional flock lock stack for transactional I/O patterns
Complex I/O patterns on file I/O can sometimes extensive nested function calls that each
manipulate flocks independently. Those nested calls can come into conflict when one call
degrades a needed lock applied by a previous call.
For instance, a pattern where a new block needs to be allocated to an opened file, but an
index of blocks must be maintained within the same file. One might have a function which
retrieves the list of block addresses from the index, and that function rationally applies
a shared flock before reading, and removes it afterwards. One might try to get an
exclusive lock on the file, then retrieve the index using the existing function. That
function, however, would end up replacing your exclusive lock with the shared lock,
potentially making it impossible to reacquire that exclusive lock depending on other
processes and their I/O.
The lock stack attempts to solve those kinds of problems by maintaining a stack of flocks,
and making sure that no new locks degrade the previous locks. In previous example, it
would notice that the stack was opened with an exclusive lock, and when the index
retrieval function attempts to apply the shared lock, it would simply upgrade that lock to
preserve the exclusive lock. Since a stack tracks each call to pflock(), once that
function attempts to release the shared lock, the lock stack would simply pop off it's
upgraded call from the stack, and make sure the preceding lock stays in place.
Another way to describe this in psuedo code:
# Enable the lock stack
PIOLOCKSTACK = 1;
sub readIdx {
pflock($file, LOCK_SH);
# ... read data
pflock($file, LOCK_UN);
# ... return data
}
sub writeIdx {
pflock($file, LOCK_EX);
# ... write data
pflock($file, LOCK_UN);
}
sub writeData {
pflock($file, LOCK_EX);
# ... write data
pflock($file, LOCK_UN);
}
sub writeTx {
pflock($file, LOCK_EX);
readIdx();
writeData();
writeIdx();
pflock($file, LOCK_UN);
}
# Execute the transaction
writeTx();
Without the lock stack, executing the transaction function would cause the following to
happen:
writeTx:
# apply LOCK_EX
# readIdx:
# apply LOCK_SH
# read data
# release all locks w/LOCK_UN
# writeData:
# apply LOCK_EX
# ERROR: any write decisions at this point based on the previous
# ERROR: index read may cause file corruption because the index
# ERROR: may have changed while this process was waiting to
# ERROR: reacquire the exclusive lock!
With the lock stack in place, however, it goes like this:
writeTx:
# apply LOCK_EX
# lock stack: (LOCK_EX)
# readIdx:
# asks for LOCK_SH, but maintains LOCK_EX
# lock stack: (LOCK_EX, LOCK_EX)
# read data
# deletes its lock from the stack, but preserves the previous lock
# lock stack: (LOCK_EX)
# writeData:
# asks for LOCK_EX
# lock stack: (LOCK_EX, LOCK_EX)
# writes data
# deletes its lock from the stack, but preserves the previous lock
# lock stack: (LOCK_EX)
# writeIdx:
# asks for LOCK_EX
# lock stack: (LOCK_EX, LOCK_EX)
# writes data
# deletes its lock from the stack, but preserves the previous lock
# lock stack: (LOCK_EX)
# release lock
# lock stack: ()
At no point was the advisory lock lost, and hence, transactional integrity was preserved
for all compliant processes.
The lock stack is off by default to allow the developer complete control over locking and
I/O patterns, but it's there to make functions easier to write without having to worry
about any locks applied outside of their code scope.
One downside of the lock stack is that affects all I/O performed via the Paranoid::IO
framework, it is not locallized to specific file handles. For that reason, one must be
confident that flocks are applied as atomically as possible throughout the code space
leveraging it.
IMPORT LISTS
This module exports the following symbols by default:
pclose pcloseAll popen preopen ptell pseek pflock
plockstat pread pnlread pwrite pappend ptruncate
The following specialized import lists also exist:
List Members
---------------------------------------------------------
all @defaults PIOBLKSIZE PIOMAXFSIZE PIOLOCKSTACK
SUBROUTINES/METHODS
PIOBLKSIZE
PIOBLKSIZE = 65536;
This lvalue function is not exported by default. It is used to determine the default
block size to read when a size is not explicitly passed. Depending on hardware and file
system parameters there might be performance gains to be had when doing default-sized
reads.
The default is 4096, which is generally a safe size for most applications.
PIOMAXFSIZE
PIOMAXFSIZE = 131072;
This lvalue function is not exported by default. It is used to determine the maximum file
size that will be read. This is not used in this module, but provided for use in
dependent modules that may want to impose file size limits, such as Paranoid::IO::Line and
others.
The default is 65536.
PIOLOCKSTACK
PIOLOCKSTACK = 1
This lvalue function is not exported by default. It is used to enable the flock lock
stack functionality in order to support transactional I/O patterns. It is disabled by
default.
popen
$fh = popen($filename);
$fh = popen($filename, $mode);
$fh = popen($filename, $mode, $perms);
$fh = popen($fh);
Returns a file handle if the file could be opened. If the mode is omitted the default is
O_CREAT | O_RDWR. File permissions (for newly created files) default to 0666 ^ UMASK.
Failures to open a file will result in an undef return value, with a text description of
the fault stored in Paranoid::ERROR.
If a file handle is passed to popen it will attempt to match it to a tracked file handle
and, if identified, take the appropriate action. If it doesn't match any tracked file
handles it will just return that file handle back to the caller.
pclose
$rv = pclose($filename);
$rv = pclose($fh);
Returns the value from close. Attempts to close a file that's already closed is
considered a success, and true value is returned. Handing it a stale file handle,
however, will be handed to the internal close, with all the expected results.
preopen
$rv = preopen();
$rv = preopen(@filenames);
$rv = preopen(@filehandles);
This checks each tracked file handle (i.e., file handles that were opened by popen) and
reopens them if necessary. This is typically only useful after a fork. It is also not
striclty necessary since every call to a function in this module does that with every
invocation, but if you have several file handles that you may not access immediately you
run the risk of the parent moving the current file position before the child gets back to
those files. You may or may not care. If you do, use this function immediately after a
fork.
Called with a list of file names means that only those files are examined and reopened.
Any failure to reopen any single file handle will result in a false return value. That
said, any failures will not interrupt the function from trying every file in the list.
pcloseAll
$rv = pcloseAll();
$rv = pcloseAll(@filenames);
$rv = pcloseAll(@filehandles);
This function returns a boolean value denoting any errors while trying to close every
tracked file handle. This function is also not strictly necessary for all the normal Perl
I/O reasons, but it's here for those that want to be explicit.
ptell
$pos = ptell($filename);
$pos = ptell($fh);
Returns the current position of the file cursor. Returns the results of sysseek, which
means that any successful seek is true, even if the cursor is at the beginning of the
file. In that instance it returns "0 but true" which is boolean true while converting to
an integer appropriately.
Any failures are returned as false or undef.
pseek
$rv = pseek($filename, $pos, $whence);
$rv = pseek($fh, $pos, $whence);
This returns the return value from sysseek. The appropriate whence values sould be one of
the SEEK_* constants as exported by Fcntl.
pflock
$rv = pflock($filename, $locktype);
$rv = pflock($fh, $locktype);
This returns the return value from flock. The appropriate lock type values should be one
of the LOCK_* constants as exported by Fcntl.
NOTE: This function essentially acts like a pass-through to the native flock function for
any file handle not opened via this module's functions.
plockstat
$lock = plockstat($filename);
This returns the last flock applied via pflock.
pread
$bytes = pread($filename, $text, $length);
$bytes = pread($filename, $text, $length, $offset);
$bytes = pread($fh, $text, $length);
$bytes = pread($fh, $text, $length, $offset);
This returns the number of bytes read, or undef on errors. If this is called prior to an
explicit popen it will default to a mode of O_RDONLY. Length defaults to PIOBLKSIZE.
pnlread
$bytes = pnlread($filename, $text, $length);
$bytes = pnlread($filename, $text, $length, $offset);
$bytes = pnlread($fh, $text, $length);
$bytes = pnlread($fh, $text, $length, $offset);
This is a wrapper function for pread that calls it with inherent file locking disabled.
It is assumed that the dev is either managing flocks directly, or they're not needed for
this application.
pwrite
$bytes = pwrite($filename, $text);
$bytes = pwrite($filename, $text, $length);
$bytes = pwrite($filename, $text, $length, $offset);
$bytes = pwrite($filename, $text, $length, $nolock);
$bytes = pwrite($fh, $text);
$bytes = pwrite($fj, $text, $length);
$bytes = pwrite($fh, $text, $length, $offset);
$bytes = pwrite($fh, $text, $length, $offset, $nolock);
This returns the number of bytes written, or undef for any critical failures. If this is
called prior to an explicit popen it uses a default mode of O_WRONLY | O_CREAT | O_TRUNC.
The optional boolean fifth argument (nolock) will bypass automatic flocks since it assumes
you're managing the lock directly.
pnlwrite
$bytes = pnlwrite($filename, $text);
$bytes = pnlwrite($filename, $text, $length);
$bytes = pnlwrite($filename, $text, $length, $offset);
$bytes = pnlwrite($fh, $text);
$bytes = pnlwrite($fj, $text, $length);
$bytes = pnlwrite($fh, $text, $length, $offset);
This is a wrapper function for pwrite that calls it with inherent file locking disabled.
It is assumed that the dev is either managing flocks directly, or they're not needed for
this application.
pappend
$bytes = pappend($filename, $text);
$bytes = pappend($filename, $text, $length);
$bytes = pappend($filename, $text, $length, $offset);
$bytes = pappend($filename, $text, $length, $offset, $nolock);
$bytes = pappend($fh, $text);
$bytes = pappend($fh, $text, $length);
$bytes = pappend($fh, $text, $length, $offset, $nolock);
This behaves identically to pwrite with the sole exception that this preserves the file
position after explicitly seeking and writing to the end of the file. The default mode
here, however, would be O_WRONLY | O_CREAT | O_APPEND for those files not explicitly
opened.
The optional boolean fifth argument (nolock) will bypass automatic flocks since it assumes
you're managing the lock directly.
pnlappend
$bytes = pnlappend($filename, $text);
$bytes = pnlappend($filename, $text, $length);
$bytes = pnlappend($filename, $text, $length, $offset);
$bytes = pnlappend($fh, $text);
$bytes = pnlappend($fj, $text, $length);
$bytes = pnlappend($fh, $text, $length, $offset);
This is a wrapper function for pappend that calls it with inherent file locking disabled.
It is assumed that the dev is either managing flocks directly, or they're not needed for
this application.
ptruncate
$rv = ptruncate($filename);
$rv = ptruncate($filename, $pos, $nolock);
$rv = ptruncate($fh);
$rv = ptruncate($fh, $pos, $nolock);
This returns the result of the internal truncate call. If called without an explicit
popen it will open the named file with the default mode of O_RDWR | O_CREAT. Omitting the
position to truncate from will result in the file being truncated at the beginning of the
file.
The optional boolean third argument (nolock) will bypass automatic flocks since it assumes
you're managing the lock directly.
pnltruncate
$rv = pnltruncate($filename);
$rv = pnltruncate($fh);
This is a wrapper function for pnltruncate that calls it with inherent file locking
disabled. It is assumed that the dev is either managing flocks directly, or they're not
needed for this application.
DEPENDENCIES
o Cwd
o Fcntl
o IO::Handle
o Paranoid
o Paranoid::Debug
o Paranoid::Input
BUGS AND LIMITATIONS
It may not always be benficial to cache file handles. You must explicitly pclose file
handles to avoid that. That said, with straight Perl you'd have to either explicitly
close the file handles or use lexical scoping, anyway. From that perspective I don't find
it onerous to do so, especially with all of the other code-saving features this module
provides.
AUTHOR
Arthur Corliss (corliss@digitalmages.com)
LICENSE AND COPYRIGHT
This software is free software. Similar to Perl, you can redistribute it and/or modify it
under the terms of either:
a) the GNU General Public License
<https://www.gnu.org/licenses/gpl-1.0.html> as published by the
Free Software Foundation <http://www.fsf.org/>; either version 1
<https://www.gnu.org/licenses/gpl-1.0.html>, or any later version
<https://www.gnu.org/licenses/license-list.html#GNUGPL>, or
b) the Artistic License 2.0
<https://opensource.org/licenses/Artistic-2.0>,
subject to the following additional term: No trademark rights to "Paranoid" have been or
are conveyed under any of the above licenses. However, "Paranoid" may be used fairly to
describe this unmodified software, in good faith, but not as a trademark.
(c) 2005 - 2021, Arthur Corliss (corliss@digitalmages.com) (tm) 2008 - 2021, Paranoid Inc.
(www.paranoid.com)