NAME

       sdr - Simple Data Recorder library

SYNOPSIS

           #include "sdr.h"

           [see below for available functions]

DESCRIPTION

       SDR is a library of functions that support the use of an abstract data recording device
       called an "SDR" ("simple data recorder") for persistent storage of data.  The SDR
       abstraction insulates software not only from the specific characteristics of any single
       data storage device but also from some kinds of persistent data storage and retrieval
       chores.  The underlying principle is that an SDR provides standardized support for user
       data organization at object granularity, with direct access to persistent user data
       objects, rather than supporting user data organization only at "file" granularity and
       requiring the user to implement access to the data objects accreted within those files.

       The SDR library is designed to provide some of the same kinds of directory services as a
       file system together with support for complex data structures that provide more
       operational flexibility than files.  (As an example of this flexibility, consider how much
       easier and faster it is to delete a given element from the middle of a linked list than it
       is to delete a range of bytes from the middle of a text file.)  The intent is to enable
       the software developer to take maximum advantage of the high speed and direct byte
       addressability of a non-volatile flat address space in the management of persistent data.
       The SDR equivalent of a "record" of data is simply a block of nominally persistent memory
       allocated from this address space.  The SDR equivalent of a "file" is a collection object.
       Like files, collections can have names, can be located by name within persistent storage,
       and can impose structure on the data items they encompass.  But, as discussed later, SDR
       collection objects can impose structures other than the strict FIFO accretion of records
       or bytes that characterizes a file.

       The notional data recorder managed by the SDR library takes the form of a single array of
       randomly accessible, contiguous, nominally persistent memory locations called a heap.
       Physically, the heap may be implemented as a region of shared memory, as a single file of
       predefined size, or both -- that is, the heap may be a region of shared memory that is
       automatically mirrored in a file.

       SDR services that manage SDR data are provided in several layers, each of which relies on
       the services implemented at lower levels:

           At the highest level, a cataloguing service enables retrieval of persistent objects by
           name.

           Services that manage three types of persistent data collections are provided for use
           both by applications and by the cataloguing service: linked lists, self-delimiting
           tables (which function as arrays that remember their own dimensions), and self-
           delimiting strings (short character arrays that remember their lengths, for speedier
           retrieval).

           Basic SDR heap space management services, analogous to malloc() and free(), enable the
           creation and destruction of objects of arbitrary type.

           Farther down the service stack are memcpy-like low-level functions for reading from
           and writing to the heap.

           Protection of SDR data integrity across a series of reads and writes is provided by a
           transaction mechanism.

       SDR persistent data are referenced in application code by Object values and Address
       values, both of which are simply displacements (offsets) within SDR address space.  The
       difference between the two is that an Object is always the address of a block of heap
       space returned by some call to sdr_malloc(), while an Address can refer to any byte in the
       address space.  That is, an Address is the SDR functional equivalent of a C pointer in
       DRAM, and some Addresses point to Objects.

       Before using SDR services, the services must be loaded to the target machine and
       initialized by invoking the sdr_initialize() function and the management profiles of one
       or more SDR's must be loaded by invoking the sdr_load_profile() function.  These steps are
       normally performed only once, at application load time.

       An application gains access to an SDR by passing the name of the SDR to the
       sdr_start_using() function, which returns an Sdr pointer.  Most other SDR library
       functions take an Sdr pointer as first argument.

       All writing to an SDR heap must occur during a transaction that was initiated by the task
       issuing the write.  Transactions are single-threaded; if task B wants to start a
       transaction while a transaction begun by task A is still in progress, it must wait until
       A's transaction is either ended or cancelled.  A transaction is begun by calling
       sdr_begin_xn().  The current transaction is normally ended by calling the sdr_end_xn()
       function, which returns an error return code value in the event that any serious SDR-
       related processing error was encountered in the course of the transaction.  Transactions
       may safely be nested, provided that every level of transaction activity that is begun is
       properly ended.

       The current transaction may instead be cancelled by calling sdr_cancel_xn(), which is
       normally used to indicate that some sort of serious SDR-related processing error has been
       encountered.  Canceling a transaction reverses all SDR update activity performed up to
       that point within the scope of the transaction -- and, if the canceled transaction is an
       inner, nested transaction, all SDR update activity performed within the scope of every
       outer transaction encompassing that transaction and every other transaction nested within
       any of those outer transactions -- provided the SDR was configured for transaction
       reversibility.  When an SDR is configured for reversibility, all heap write operations
       performed during a transaction are recorded in a log file that is retained until the end
       of the transaction.  Each log file entry notes the location at which the write operation
       was performed, the length of data written, and the content of the overwritten heap bytes
       prior to the write operation.  Canceling the transaction causes the log entries to be read
       and processed in reverse order, restoring all overwritten data.  Ending the transaction,
       on the other hand, simply causes the log to be discarded.

       If a log file exists at the time that the profile for an SDR is loaded (typically during
       application initialization), the transaction that was being logged is automatically
       canceled and reversed.  This ensures that, for example, a power failure that occurs in the
       middle of a transaction will never wreck the SDR's data integrity: either all updates
       issued during a given transaction are reflected in the current database content or none
       are.

       As a further measure to protect SDR data integrity, an SDR may additionally be configured
       for object bounding.  When an SDR is configured to be "bounded", every heap write
       operation is restricted to the extent of a single object allocated from heap space; that
       is, it's impossible to overwrite part of one object by writing beyond the end of another.
       To enable the library to enforce this mechanism, application code is prohibited from
       writing anywhere but within the extent of an object that either (a) was allocated from
       managed heap space during the same transaction (directly or indirectly via some collection
       management function) or (b) was staged -- identified as an update target -- during the
       same transaction (again, either directly or via some collection management function).

       Note that both transaction reversibility and object bounding consume processing cycles and
       inhibit performance to some degree.  Determining the right balance between operational
       safety and processing speed is left to the user.

       Note also that, since SDR transactions are single-threaded, they can additionally be used
       as a general mechanism for simply implementing "critical sections" in software that is
       already using SDR for other purposes: the beginning of a transaction marks the start of
       code that can't be executed concurrently by multiple tasks.  To support this use of the
       SDR transaction mechanism, the additional transaction termination function sdr_exit_xn()
       is provided.  sdr_exit_xn() simply ends a transaction without either signaling an error or
       checking for errors.  Like sdr_cancel_xn(), sdr_exit_xn() has no return value; unlike
       sdr_cancel_xn(), it assures that ending an inner, nested transaction does not cause the
       outer transaction to be aborted and backed out.  But this capability must be used
       carefully: the protection of SDR data integrity requires that transactions which are ended
       by sdr_exit_xn() must not encompass any SDR update activity whatsoever.

       The heap space management functions of the SDR library are adapted directly from the
       Personal Space Management (psm) function library.  The manual page for psm(3) explains the
       algorithms used and the rationale behind them.  The principal difference between PSM
       memory management and SDR heap management is that, for performance reasons, SDR reserves
       the "small pool" for its own use only; all user data space is allocated from the "large
       pool", via the sdr_malloc() function.

   RETURN VALUES AND ERROR HANDLING
       Whenever an SDR function call fails, a diagnostic message explaining the failure of the
       function is recorded in the error message pool managed by the "platform" system (see the
       discussion of putErrmsg() in platform(3)).

       The failure of any function invoked in the course of an SDR transaction causes all
       subsequent SDR activity in that transaction to fail immediately.  This can streamline SDR
       application code somewhat: it may not be necessary to check the return value of every SDR
       function call executed during a transaction.  If the sdr_end_xn() call returns zero, all
       updates performed during the transaction must have succeeded.

SYSTEM ADMINISTRATION FUNCTIONS

       int sdr_initialize(int wmSize, char *wmPtr, int wmKey, char *wmName)
           Initializes the SDR system.  sdr_initialize() must be called once every time the
           computer on which the system runs is rebooted, before any call to any other SDR
           library function.

           This function attaches to a pool of shared memory, managed by PSM (see psm(3), that
           enables SDR library operations.  If the SDR system is to access a common pool of
           shared memory with one or more other systems, the key of that shared memory segment
           must be provided in wmKey and the PSM partition name associated with that memory
           segment must be provided in wmName; otherwise wmKey must be zero and wmName must be
           NULL, causing sdr_initialize() to assign default values.  If a shared memory segment
           identified by the effective value of wmKey already exists, then wmSize may be zero and
           the value of wmPtr is ignored.  Otherwise the size of the shared memory pool must be
           provided in wmSize and a new shared memory segment is created in a manner that is
           dependent on wmPtr: if wmPtr is NULL then wmSize bytes of shared memory are
           dynamically acquired, allocated, and assigned to the newly created shared memory
           segment; otherwise the memory located at wmPtr is assumed to have been pre-allocated
           and is merely assigned to the newly created shared memory segment.

           sdr_initialize() also creates a semaphore to serialize access to the SDR system's
           private array of SDR profiles.

           Returns 0 on success, -1 on any failure.

       void sdr_wm_usage(PsmUsageSummary *summary)
           Loads summary with a snapshot of the usage of the SDR system's private working memory.
           To print the snapshot, use psm_report().  (See psm(3).)

       void sdr_shutdown( )
           Ends all access to all SDRs (see sdr_stop_using()), detaches from the SDR system's
           working memory (releasing the memory if it was dynamically allocated by
           sdr_initialize()), and destroys the SDR system's private semaphore.  After
           sdr_shutdown(), sdr_initialize() must be called again before any call to any other SDR
           library function.

DATABASE ADMINISTRATION FUNCTIONS

       int sdr_load_profile(char *name, int configFlags, long heapWords, int memKey, char
       *pathName, char *restartCmd, unsigned int restartLatency)
           Loads the profile for an SDR into the system's private list of SDR profiles.  Although
           SDRs themselves are persistent, SDR profiles are not: in order for an application to
           access an SDR, sdr_load_profile() must have been called to load the profile of the SDR
           since the last invocation of sdr_initialize().

           name is the name of the SDR, required for any subsequent sdr_start_using() call.

           configFlags specifies the configuration of the SDR, the bitwise "or" of some
           combination of the following:

           SDR_IN_DRAM
               SDR is implemented as a region of shared memory.

           SDR_IN_FILE
               SDR is implemented as a file.

           SDR_REVERSIBLE
               SDR transactions are logged and are reversed if canceled.

           SDR_BOUNDED
               Heap updates are not allowed to cross object boundaries.

           heapWords specifies the size of the heap in words; word size depends on machine
           architecture, i.e., a word is 4 bytes on a 32-bit machine, 8 bytes on a 64-bit
           machine.  Note that each SDR prepends to the heap a "map" of predefined, fixed size.
           The total amount of space occupied by an SDR in memory and/or in a file is the sum of
           the size of the map plus the product of word size and heapWords.

           memKey is ignored if configFlags does not include SDR_IN_DRAM.  It should normally be
           SM_NO_KEY, causing the shared memory region for the SDR to be allocated dynamically
           and shared using a dynamically selected shared memory key.  If specified, memKey must
           be a shared memory key identifying a pre-allocated region of shared memory whose
           length is equal to the total SDR size, shared via the indicated key.

           pathName is ignored if configFlags includes neither SDR_REVERSIBLE nor SDR_IN_FILE.
           It is the fully qualified name of the directory into which the SDR's log file and/or
           database file will be written.  The name of the log file (if any) will be
           "<sdrname>.sdrlog".  The name of the database file (if any) will be "<sdrname>.sdr";
           this file will be automatically created and filled with zeros if it does not exist at
           the time the SDR's profile is loaded.

           If a cleanup task must be run whenever a transaction is reversed, the command to
           execute this task must be provided in restartCmd and the number of seconds to wait for
           this task to finish before resuming operations must be provided in restartLatency.  If
           restartCmd is NULL or restartLatency is zero then no cleanup task will be run upon
           transaction reversal.

           Returns 0 on success, -1 on any error.

       int sdr_reload_profile(char *name, int configFlags, long heapWords, int memKey, char
       *pathName, char *restartCmd, unsigned int restartLatency)
           For use when the state of an SDR is thought to be inconsistent, perhaps due to crash
           of a program that had a transaction open.  Unloads the profile for the SDR, forcing
           the reversal of any transaction that is currently in progress when the SDR's profile
           is re-loaded.  Then calls sdr_load_profile() to re-load the profile for the SDR.  Same
           return values as sdr_load_profile.

       Sdr sdr_start_using(char *name)
           Locates SDR profile by name and returns a handle that can be used for all functions
           that operate on that SDR.  On any failure, returns NULL.

       char *sdr_name(Sdr sdr)
           Returns the name of the sdr.

       long sdr_heap_size(Sdr sdr)
           Returns the total size of the SDR heap, in bytes.

       void sdr_stop_using(Sdr sdr)
           Terminates access to the SDR via this handle.  Other users of the SDR are not
           affected.  Frees the Sdr object.

       void sdr_abort(Sdr sdr)
           Terminates the task.  In flight configuration, also terminates all use of the SDR
           system by all tasks.

       void sdr_destroy(Sdr sdr)
           Ends all access to this SDR, unloads the SDR's profile, and erases the SDR from memory
           and file system.

DATABASE TRANSACTION FUNCTIONS

       void sdr_begin_xn(Sdr sdr)
           Initiates a transaction.  Note that transactions are single-threaded; any task that
           calls sdr_begin_xn() is suspended until all previously requested transactions have
           been ended or canceled.

       int sdr_in_xn(Sdr sdr)
           Returns 1 if called in the course of a transaction, 0 otherwise.

       void sdr_exit_xn(Sdr sdr)
           Simply abandons the current transaction, ceasing the calling task's lock on ION.  Must
           not be used if any database modifications were performed during the transaction;
           sdr_end_xn() must be called instead, to commit those modifications.

       void sdr_cancel_xn(Sdr sdr)
           Cancels the current transaction.  If reversibility is enabled for the SDR, canceling a
           transaction reverses all heap modifications performed during that transaction.

       int sdr_end_xn(Sdr sdr)
           Ends the current transaction.  Returns 0 if the transaction completed without any
           error; returns -1 if any operation performed in the course of the transaction failed,
           in which case the transaction was automatically canceled.

DATABASE I/O FUNCTIONS

       void sdr_read(Sdr sdr, char *into, Address from, int length)
           Copies length characters at from (a location in the indicated SDR) to the memory
           location given by into.  The data are copied from the shared memory region in which
           the SDR resides, if any; otherwise they are read from the file in which the SDR
           resides.

       void sdr_peek(sdr, variable, from)
           sdr_peek() is a macro that uses sdr_read() to load variable from the indicated address
           in the SDR database; the size of variable is used as the number of bytes to copy.

       void sdr_write(Sdr sdr, Address into, char *from, int length)
           Copies length characters at from (a location in memory) to the SDR heap location given
           by into.  Can only be performed during a transaction, and if the SDR is configured for
           object bounding then heap locations into through (into + (length - 1)) must be within
           the extent of some object that was either allocated or staged within the same
           transaction.  The data are copied both to the shared memory region in which the SDR
           resides, if any, and also to the file in which the SDR resides, if any.

       void sdr_poke(sdr, into, variable)
           sdr_poke() is a macro that uses sdr_write() to store variable at the indicated address
           in the SDR database; the size of variable is used as the number of bytes to copy.

       char *sdr_pointer(Sdr sdr, Address address)
           Returns a pointer to the indicated location in the heap - a "heap pointer" - or NULL
           if the indicated address is invalid.  NOTE that this function cannot be used if the
           SDR does not reside in a shared memory region.

           Providing an alternative to using sdr_read() to retrieve objects into local memory,
           sdr_pointer() can help make SDR-based applications run very quickly, but it must be
           used WITH GREAT CAUTION!  Never use a direct pointer into the heap when not within a
           transaction, because you will have no assurance at any time that the object pointed to
           by that pointer has not changed (or is even still there).  And NEVER de-reference a
           heap pointer in order to write directly into the heap: this makes transaction reversal
           impossible.  Whenever writing to the SDR, always use sdr_write().

       Address sdr_address(Sdr sdr, char *pointer)
           Returns the address within the SDR heap of the indicated location, which must be (or
           be derived from) a heap pointer as returned by sdr_pointer().  Returns zero if the
           indicated location is not greater than the start of the heap mirror.  NOTE that this
           function cannot be used if the SDR does not reside in a shared memory region.

       void sdr_get(sdr, variable, heap_pointer)
           sdr_get() is a macro that uses sdr_read() to load variable from the SDR address given
           by heap_pointer; heap_pointer must be (or be derived from) a heap pointer as returned
           by sdr_pointer().  The size of variable is used as the number of bytes to copy.

       void sdr_set(sdr, heap_pointer, variable)
           sdr_set() is a macro that uses sdr_write() to store variable at the SDR address given
           by heap_pointer; heap_pointer must be (or be derived from) a heap pointer as returned
           by sdr_pointer().  The size of variable is used as the number of bytes to copy.

HEAP SPACE MANAGEMENT FUNCTIONS

       Object sdr_malloc(Sdr sdr, unsigned long size)
           Allocates a block of space from the of the indicated SDR's heap.  size is the size of
           the block to allocate; the maximum size is 1/2 of the maximum address space size
           (i.e., 2G for a 32-bit machine).  Returns block address if successful, zero if block
           could not be allocated.

       Object sdr_insert(Sdr sdr, char *from, unsigned long size)
           Uses sdr_malloc() to obtain a block of space of size size and, if this allocation is
           successful, uses sdr_write() to copy size bytes of data from memory at from into the
           newly allocated block.  Returns block address if successful, zero if block could not
           be allocated.

       Object sdr_stow(sdr, variable)
           sdr_stow() is a macro that uses sdr_insert() to insert a copy of variable into the
           database.  The size of variable is used as the number of bytes to copy.

       int sdr_object_length(Sdr sdr, Object object)
           Returns the number of bytes of heap space allocated to the application data at object.

       void sdr_free(Sdr sdr, Object object)
           Frees for subsequent re-allocation the heap space occupied by object.

       void sdr_stage(Sdr sdr, char *into, Object from, int length)
           Like sdr_read(), this function will copy length characters at from (a location in the
           heap of the indicated SDR) to the memory location given by into.  Unlike sdr_get(),
           sdr_stage() requires that from be the address of some allocated object, not just any
           location within the heap.  sdr_stage(), when called from within a transaction,
           notifies the SDR library that the indicated object may be updated later in the
           transaction; this enables the library to retrieve the object's size for later
           reference in validating attempts to write into some location within the object.  If
           length is zero, the object's size is privately retrieved by SDR but none of the
           object's content is copied into memory.

       long sdr_unused(Sdr sdr)
           Returns number of bytes of heap space not yet allocated to either the large or small
           objects pool.

       void sdr_usage(Sdr sdr, SdrUsageSummary *summary)
           Loads the indicated SdrUsageSummary structure with a snapshot of the SDR's usage
           status.  SdrUsageSummary is defined by:

               typedef struct
               {
                       char            sdrName[MAX_SDR_NAME + 1];
                       unsigned int    sdrSize;
                       unsigned int    smallPoolSize;
                       unsigned int    smallPoolFreeBlockCount[SMALL_SIZES];
                       unsigned int    smallPoolFree;
                       unsigned int    smallPoolAllocated;
                       unsigned int    largePoolSize;
                       unsigned int    largePoolFreeBlockCount[LARGE_ORDERS];
                       unsigned int    largePoolFree;
                       unsigned int    largePoolAllocated;
                       unsigned int    unusedSize;
               } SdrUsageSummary;

       void sdr_report(SdrUsageSummary *summary)
           Sends to stdout a printed summary of the SDR's usage status.

       int sdr_heap_depleted(Sdr sdr)
           A Boolean function: returns 1 if the total available space in the SDR's heap (small
           pool free, large pool free, and unused) is less than 1/16 of the total size of the
           heap.  Otherwise returns zero.

HEAP SPACE USAGE TRACING

       If SDR_TRACE is defined at the time the SDR source code is compiled, the system includes
       built-in support for simple tracing of SDR heap space usage: heap space allocations are
       logged, and heap space deallocations are matched to logged allocations, "closing" them.
       This enables heap space leaks and some other kinds of SDR heap access problems to be
       readily investigated.

       int sdr_start_trace(Sdr sdr, int traceLogSize, char *traceLogAddress)
           Begins an episode of SDR heap space usage tracing.  traceLogSize is the number of
           bytes of shared memory to use for trace activity logging; the frequency with which
           "closed" trace log events must be deleted will vary inversely with the amount of
           memory allocated for the trace log.  traceLogAddress is normally NULL, causing the
           trace system to allocate traceLogSize bytes of shared memory dynamically for trace
           logging; if non-NULL, it must point to traceLogSize bytes of shared memory that have
           been pre-allocated by the application for this purpose.  Returns 0 on success, -1 on
           any failure.

       void sdr_print_trace(Sdr sdr, int verbose)
           Prints a cumulative trace report and current usage report for sdr.  If verbose is
           zero, only exceptions (notably, trace log events that remain open -- potential SDR
           heap space leaks) are printed; otherwise all activity in the trace log is printed.

       void sdr_clear_trace(Sdr sdr)
           Deletes all closed trace log events from the log, freeing up memory for additional
           tracing.

       void sdr_stop_trace(Sdr sdr)
           Ends the current episode of SDR heap space usage tracing.  If the shared memory used
           for the trace log was allocated by sdr_start_trace(), releases that shared memory.

CATALOGUE FUNCTIONS

       The SDR catalogue functions are used to maintain the catalogue of the names, types, and
       addresses of objects within an SDR.  The catalogue service includes functions for
       creating, deleting and finding catalogue entries and a function for navigating through
       catalogue entries sequentially.

       void sdr_catlg(Sdr sdr, char *name, int type, Object object)
           Associates object with name in the indicated SDR's catalogue and notes the type that
           was declared for this object.  type is optional and has no significance other than
           that conferred on it by the application.

           The SDR catalogue is flat, not hierarchical like a directory tree, and all names must
           be unique.  The length of name is limited to 15 characters.

       Object sdr_find(Sdr sdr, char *name, int *type)
           Locates the Object associated with name in the indicated SDR's catalogue and returns
           its address; also reports the catalogued type of the object in *type if type is non-
           NULL.  Returns zero if no object is currently catalogued under this name.

       void sdr_uncatlg(Sdr sdr, char *name)
           Dissociates from name whatever object in the indicated SDR's catalogue is currently
           catalogued under that name.

       Object sdr_read_catlg(Sdr sdr, char *name, int *type, Object *object, Object
       previous_entry)
           Used to navigate through catalogue entries sequentially.  If previous_entry is zero,
           reads the first entry in the indicated SDR's catalogue; otherwise, reads the next
           catalogue entry following the one located at previous_entry.  In either case, returns
           zero if no such catalogue entry exists; otherwise, copies that entry's name, type, and
           catalogued object address into name, *type, and *object, and then returns the address
           of the catalogue entry (which may be used as previous_entry in a subsequent call to
           sdr_read_catlg()).

USER'S GUIDE

       Compiling an SDR application
           Just be sure to "#include "sdr.h"" at the top of each source file that includes any
           SDR function calls.

           For UNIX applications, link with "-lsdr".

       Loading an SDR application (VxWorks)
               ld < "libsdr.o"

           After the library has been loaded, you can begin loading SDR applications.

SEE ALSO

       sdrlist(3), sdrstring(3), sdrtable(3)