Provided by: libbiblio-isis-perl_0.24-1.4_all bug

NAME

       CDS/ISIS manual appendix F, G and H

DESCRIPTION

       This is partial scan of CDS/ISIS manual (appendix F, G and H, pages 257-272) which is than
       converted to text using OCR and proofread.  However, there might be mistakes, and any
       corrections sent to "dpavlin@rot13.org" will be greatly appreciated.

       This digital version is made because current version available in ditial form doesn't
       contain details about CDS/ISIS file format and was essential in making Biblio::Isis
       module.

       This extract of manual has been produced in compliance with section (d) of WinIsis LICENCE
       for receiving institution/person which say:

        The receiving institution/person may:

        (d) Print/reproduce the CDS/ISIS manuals or portions thereof,
            provided that such copies reproduce the copyright notice;

CDS/ISIS Files

       This section describes the various files of the CDS/ISIS system, the file naming
       conventions and the file extensions used for each type of file. All CDS/ISIS files have
       standard names as follows:

         nnnnnn.eee

       where:

       "nnnnnn"  is the file name (all file names, except program names, are limited to a maximum
                 of 6 characters)

       ".eee"    is the file extension identifying a particular type of file.

       Files marked with "*" are ASCII files which you may display or print. The other files are
       binary files.

   A. System files
       System files are common to all CDS/ISIS users and include the various executable programs
       as well as system menus, worksheets and message files provided by Unesco as well as
       additional ones which you may create.

       CDS/ISIS Program

       The name of the program file, as supplied by Unesco is

         ISIS.EXE

       Depending on the release and/or target computer, there may also be one or more overlay
       files. These, if present, have the extension "OVL".  Check the contents of your system
       diskettes or tape to see whether overlay files are present.

       System menus and worksheets

       All system menus and worksheets have the file extension FMT and the names are built as
       follows:

         pctnnn.FMT

       where:

       "p"       is the page number (A for the first page, B for the second, etc.)

       "c"       is the language code (e.g. E for English), which must be one of those provided
                 for in the language selection menu xXLNG.

       "t"       is X for menus and Y for system worksheets

       "nnn"     is a unique identifier

       For example the full name of the English version of the menu xXGEN is "AEXGEN.FMT".

       The page number is transparent to the CDS/ISIS user. Like the file extension the page
       number is automatically provided by the system.  Therefore when a CDS/ISIS program prompts
       you to enter a menu or worksheet name you must not include the page number. Furthermore as
       file names are restricted to 6 characters, menus and worksheets names may not be longer
       than 5 characters.

       System menus and worksheets may only have one page.

       The language code is mandatory for system menus and standard system worksheets. For
       example if you want to link a HELP menu to the system menu EXGEN, its name must begin with
       the letter E.

       The X convention is only enforced for standard system menus. It is a good practice,
       however, to use the same convention for menus that you create, and to avoid creating
       worksheets (including data entry worksheets) with X in this position, that is with names
       like xXxxx.

       Furthermore, if a data base name contains X or Y in the second position, then the
       corresponding data entry worksheets will be created in the system worksheet directory
       (parameter 2 of "SYSPAR.PAR") rather then the data base directory. Although this will not
       prevent normal operation of the data base, it is not recommended.

       System messages files

       System messages and prompts are stored in standard CDS/ISIS data bases.  All corresponding
       data base files (see below) are required when updating a message file, but only the Master
       file is used to display messages.

       There must be a message data base for each language supported through the language
       selection menu xXLNG.

       The data base name assigned to message data bases is xMSG (where x is the language code).

       System tables

       System tables are used by CDS/ISIS to define character sets. Two are required at present:

       "ISISUC.TAB"*
           defines lower to upper-case translation

       "ISISAC.TAB"*
           defines the alphabetic characters.

       System print and work files

       Certain CDS/ISIS print functions do not send the output directly to the printer but store
       it on a disk file from which you may then print it at a convenient time. These files have
       all the file extension "LST" and are reused each time the corresponding function is
       executed.

       In addition CDS/ISIS creates temporary work files which are normally automatically
       discarded at the end of the session. If the session terminates abnormally, however, they
       will not be deleted. A case of abnormal termination would be a power failure while you are
       using a CDS/ISIS program. Also these files, however, are reused each time, so that you do
       not normally need to delete them manually. Work files all have the extension "TMP".

       The print and work files created by CDS/ISIS are given below:

       "IFLIST.LST"*
           Inverted file listing file (produced by ISISINV)

       "WSLIST.LST"*
           Worksheet/menu listing file (produced by ISISUTL)

       "xMSG.LST"*
           System messages listing file (produced by ISISUTL)

       "x.LST"*
           Printed output (produced by ISISPRT when printing no print file name is supplied)

       "SORTIO.TMP"
           Sort work file 1

       "SORTII.TMP"
           Sort work file 2

       "SORTI2.TMP"
           Sort work file 3

       "SORTI3.TMP"
           Sort work file 4

       "SORT20.TMP"
           Sort work file 5

       "SORT2I.TMP"
           Sort work file 6

       "SORT22.TMP"
           Sort work file 7

       "SORT23.TMP"
           Sort work file 8

       "TRACE.TMP"*
           Trace file created by certain programs

       "ATSF.TMP"
           Temporary storage for hit lists created during retrieval

       "ATSQ.TMP"
           Temporary storage for search expressions

   B. Data Base files
       1.  mandatory files, which must always be present.  These are normally established when
           the data base is defined by means of the ISISDEF services and should never be deleted;

       2.  auxiliary files created by the system whenever certain functions are performed.  These
           can periodically be deleted when they are no longer needed.

       3.  user files created by the data base user (such as display formats), which are fully
           under the user's responsibility.

       Each data base consists of a number of physically distinct files as indicated below. There
       are three categories of data base files:

       In the following description "xxxxxx" is the 1-6 character data base name.

       Mandatory data base files

       "xxxxxx.FDT"*
           Field Definition Table

       "xxxxxx.FST"*
           Field Select Table for Inverted file

       "xxxxxx.FMT"*
           Default data entry worksheet (where p is the page number).

           Note that the data base name is truncated to 5 characters if necessary

       "xxxxxx.PFT"*
           Default display format

       "xxxxxx.MST"
           Master file

       "xxxxxx.XRF"
           Crossreference file (Master file index)

       "xxxxxx.CNT"
           B*tree (search term dictionary) control file

       "xxxxxx.N01"
           B*tree Nodes (for terms up to 10 characters long)

       "xxxxxx.L01"
           B*tree Leafs (for terms up to 10 characters long)

       "xxxxxx.N02"
           B*tree Nodes (for terms longer than 10 characters)

       "xxxxxx.L02"
           B*tree Leafs (for terms longer than 10 characters)

       "xxxxxx.IFP"
           Inverted file postings

       "xxxxxx.ANY"*
           ANY file

       Auxiliary files

       "xxxxx.STW"*
           Stopword file used during inverted file generation

       "xxxxxx.LN1"*
           Unsorted Link file (short terms)

       "xxxxxx.LN2"*
           Unsorted Link file (long terms)

       "xxxxxx.LKl"*
           Sorted Link file (short terms)

       "xxxxxx.LK2"*
           Sorted Link file (long terms)

       "xxxxxx.BKP"
           Master file backup

       "xxxxxx.XHF"
           Hit file index

       "xxxxxx.HIT"
           Hit file

       "xxxxxx.SRT"*
           Sort convertion table (see "Uppercase conversion table (1SISUC.TAB)" on page 227)

       User files

       "yyyyyy.FST"*
           Field Select tables used for sorting

       "yyyyyy.PFT"*
           Additional display formats

       "yyyyyy.FMT"*
           Additional data entry worksheets

       "yyyyyy.STW"*
           Additional stopword files

       "yyyyyy.SAV"
           Save files created during retrieval

       The name of user files is fully under user control. However, in order to avoid possible
       name conflicts it is advisable to establish some standard conventions to be followed by
       all CDS/ISIS users at a given site, such as for example to define "yyyyyy" as follows:

         xxxyyy

       where:

       "xxx"
           is a data base identifier (which could be the first three letters of the data base
           name if no two data bases names are allowed to begin with the same three letters)

       "yyy"
           a user chosen name.

Master file structure and record format

   A. Master file record format
       The Master record is a variable length record consisting of three sections: a fixed length
       leader; a directory; and the variable length data fields.

       Leader format

       The leader consists of the following 7 integers (fields marked with * are 31-bit signed
       integers):

       "MFN"*
           Master file number

       "MFRL"
           Record length (always an even number)

       "MFBWB"*
           Backward pointer - Block number

       "MFBWP"
           Backward pointer - Offset

       "BASE"
           Offset to variable fields (this is the combined length of the Leader and Directory
           part of the record, in bytes)

       "NVF"
           Number of fields in the record (i.e. number of directory entries)

       "STATUS"
           Logical deletion indicator (0=record active; 1=record marked for deletion)

       "MFBWB" and "MFBWP" are initially set to 0 when the record is created. They are
       subsequently updated each time the record itself is updated (see below).

       Directory format

       The directory is a table indicating the record contents. There is one directory entry for
       each field present in, the record (i.e. the directory has exactly NVF entries). Each
       directory entry consists of 3 integers:

       "TAG"
           Field Tag

       "POS"
           Offset to first character position of field in the variable field section (the first
           field has "POS=0")

       "LEN"
           Field length in bytes

       The total directory length in bytes is therefore "6*NVF"; the "BASE" field in the leader
       is always: "18+6*NVF".

       Variable fields

       This section contains the data fields (in the order indicated by the directory). Data
       fields are placed one after the other, with no separating characters.

   B. Control record
       The first record in the Master file is a control record which the system maintains
       automatically. This is never accessible to the ISIS user. Its contents are as follows
       (fields marked with "*" are 31-bit signed integers):

       "CTLMFN"*
           always 0

       "NXTMFN"*
           MFN to be assigned to the next record created in the data base

       "NXTMFB"*
           Last block number allocated to the Master file (first block is 1)

       "NXTMFP"
           Offset to next available position in last block

       "MFTYPE"
           always 0 for user data base file (1 for system message files)

       (the last four fields are used for statistics during backup/restore).

   C. Master file block format
       The Master file records are stored consecutively, one after the other, each record
       occupying exactly "MFRL" bytes. The file is stored as physical blocks of 512 bytes. A
       record may begin at any word boundary between 0-498 (no record begins between 500-510) and
       may span over two or more blocks.

       As the Master file is created and/or updated, the system maintains an index indicating the
       position of each record. The index is stored in the Crossreference file (".XRF")

   D. Crossreference file
       The "XRF" file is organized as a table of pointers to the Master file.  The first pointer
       corresponds to MFN 1, the second to MFN 2, etc.

       Each pointer consists of two fields:

       "RECCNT"*
       "MFCXX1"*
       "MFCXX2"*
       "MFCXX3"*
       "XRFMFB"
           (21 bits) Block number of Master file block containing the record

       "XRFMFP"
           (11 bits) Offset in block of first character position of Master record (first block
           position is 0)

       which are stored in a 31-bit signed integer (4 bytes) as follows:

         pointer = XRFMFB * 2048 + XRFMFP

       (giving therefore a maximum Master file size of 500 Megabytes).

       Each block of the "XRF" file is 512 bytes and contains 127 pointers. The first field in
       each block ("XRFPOS") is a 31-bit signed integer whose absolute value is the "XRF" block
       number. A negative "XRFPOS" indicates the last block.

       Deleted records are indicated as follows:

       "XRFMFB < 0" and "XRFMFP > 0"
           logically deleted record (in this case "ABS(XRFMFB)" is the correct block pointer and
           "XRFMFP" is the offset of the record, which can therefore still be retrieved)

       "XRFMFB = -1" and "XRFMFP = 0"
           physically deleted record

       "XRFMFB = 0" and "XRFMFP = 0"
           inexistent record (all records beyond the highest "MFN" assigned in the data base)

   E. Master file updating technique
       Creation of new records

       New records are always added at the end of the Master file, at the position indicated by
       the fields "NXTMFB"/"NXTMFP" in the Master file control record. The "MFN" to be assigned
       is also obtained from the field "NXTMFN" in the control record.

       After adding the record, "NXTMFN" is increased by 1 and "NXTMFB"/"NXTMFP" are updated to
       point to the next available position. In addition a new pointer is created in the "XRF"
       file and the "XRFMFP" field corresponding to the record is increased by 1024 to indicate
       that this is a new record to be inverted (after the inversion of the record 1024 is
       subtracted from "XRFMFP").

       Update of existing records

       Whenever you update a record (i.e., you call it in data entry and exit with option X from
       the editor) the system writes the record back to the Master file. Where it is written
       depends on the status of the record when it was initially read.

       There was no inverted file update pending for the record

       This condition is indicated by the following:

       On "XRF" "XRFMFP < 512" and

       On "MST" "MFBWB = 0" and "MFBWP = 0"

       In this case, the record is always rewritten at the end of the Master file (as if it were
       a new record) as indicated by "NXTMFB"/"NXTMFP" in the control record. In the new version
       of the record "MFBWB"/"MFBWP" are set to point to the old version of the record, while in
       the "XRF" file the pointer points to the new version. In addition 512 is added to "XRFMFP"
       to indicate that an inverted file update is pending. When the inverted file is updated,
       the old version of the record is used to determine the postings to be deleted and the new
       version is used to add the new postings. After the update of the Inverted file, 512 is
       subtracted from "XRFMFP", and "MFBWB"/"MFBWP" are reset to 0.

       An inverted file update was pending

       This condition is indicated by the following:

       On "XRF" "XRFMFP > 512" and

       On "MST" "MFBWB > 0"

       In this case "MFBWB"/"MFBWP" point to the version of the record which is currently
       reflected in the Inverted file. If possible, i.e. if the record length was not increased,
       the record is written back at its original location, otherwise it is written at the end of
       the file. In both cases, "MFBWB"/"MFBWP" are not changed.

       Deletion of records

       Record deletion is treated as an update, with the following additional markings:

       On "XRF" "XRFMFB" is negative

       On "MST" "STATUS" is set to 1

   F. Master file reorganization
       As indicated above, as Master file records are updated the "MST" file grows in size and
       there will be lost space in the file which cannot be used. The reorganization facilities
       allow this space to be reclaimed by recompacting the file.

       During the backup phase a Master file backup file is created (".BKP").  The structure and
       format of this file is the same as the Master file (".MST"), except that a Crossreference
       file is not required as all the records are adjacent. Records marked for deletion are not
       backed up.  Because only the latest copy of each record is backed up, the system does not
       allow you to perform a backup whenever an Inverted file update is pending for one or more
       records.

       During the restore phase the backup file is read sequentially and the program recreates
       the "MST" and "XRF" file. At this point alt records which were marked for logical deletion
       (before the backup) are now marked as physically deleted (by setting "XRFMFB = -1" and
       "XRFMFP = 0".  Deleted records are detected by checking holes in the "MFN" numbering.

Inverted file structure and record formats

   A. Introduction
       The CDS/ISIS Inverted file consists of six physical files, five of which contain the
       dictionary of searchable terms (organized as a B*tree) and the sixth contains the list of
       postings associated with each term. In order to optimize disk storage, two separate
       B*trees are maintained, one for terms of up to 10 characters (stored in files
       ".N01"/".L01") and one for terms longer than 10 characters, up to a maximum of 30
       characters (stored in files ".N02"/".L02"). The file "CNT" contains control fields for
       both B*trees. In each B*tree the file ".N0x" contains the nodes of the tree and the ".L0x"
       file contains the leafs. The leaf records point to the postings file ".IFP".

       The relationship between the various files is schematically represented in Figure 67.

       The physical relationship between these six files is a pointer, which represents the
       relative address of the record being pointed to. A relative address is the ordinal record
       number of a record in a given file (i.e. the first record is record number 1, the second
       is record number 2, etc.). The file ".CNT" points to the file ".N0x", ".N0x" points to
       ".L0x", and ".L0x" points to ".IFP". Because the ".IFP" is a packed file, the pointer from
       ".L0x" to ".IFP" has two components: the block number and the offset within the block,
       each expressed as an integer.

   B. Format of ".CNT" file
       This file contain two 26-byte fixed length records (one for each B*tree) each containing
       10 integers as follows (fields marked with * are 31-bit signed integers):

       "IDTYPE"
           B*tree type (1 for ".N01"/".L01", 2 for ".N02"/".L02")

       "ORDN"
           Nodes order (each ".N0x" record contains at most "2*ORDN" keys)

       "ORDF"
           Leafs order (each ".L0x" record contains at most "2*ORDF" keys)

       "N" Number of memory buffers allocated for nodes

       "K" Number of buffers allocated to lst level index ("K < N")

       "LIV"
           Current number of index levels

       "POSRX"*
           Pointer to Root record in ".N0x"

       "NMAXPOS"*
           Next available position in ".N0x" file

       "FMAXPOS"*
           Next available position in ".L0x" file

       "ABNORMAL"
           Formal B*tree normality indicator (0 if B*tree is abnormal, 1 if B*tree is normal). A
           B*tree is abnormal if the nodes file ".N0x" contains only the Root.

       "ORDN", "ORDF", "N" and "K" are fixed for a given generated system.  Currently these
       values are set as follows:

       "ORDN = 5"; "ORDF = 5"; "N = 15"; "K = 5" for both B*trees

                         +--------------+
                         | Root address |
                         +-------|------+
                                 |                          .CNT file
                                 |                      -------------
                                 |                          .N0x file
                     +-----------V--------+
                     | Key1 Key2 ... Keyn |                   Root
                     +---|-------------|--+
                         |             |
                   +-----+             +------+
                   |                          |
        +----------V----------+     +---------V----------+ 1st level
        | Key1  Key2 ... Keyn | ... | Key1 Key2 ... Keyn |   index
        +--|------------------+     +-----------------|--+
           |                                          :
           :                                  +-------+
           |                                  |
        +--V------------------+     +---------V----------+ last level
        | Key1  Key2 ... Keyn | ... | Key1 Key2 ... Keyn |   index
        +---------|-----------+     +---------|----------+
                  |                           |
                  |                           |         -------------
                  |                           |             .L0x file
        +---------V-----------+     +---------V----------+
        | Key1  Key2 ... Keyn | ... | Key1 Key2 ... Keyn |
        +--|------------------+     +--------------------+
           |
           |                                            -------------
           |                                                .IPF file
        +--V----------------------------------+
        | P1  P2  P3 ..................... Pn |
        +-------------------------------------+

       Figure 67: Inverted file structure

       The other values are set as required when the B*trees are generated.

   C. Format of ".N0x" files
       These files contain the indexes) of the dictionary of searchable terms (".N01" for terms
       shorter than 11 characters and ".N02" for terms longer than 10 characters). The ".N0x"
       file records have the following format (fields marked with * are 31-bit signed integers):

       "POS"*
           an integer indicating the relative record number (1 for the first record, 2 for the
           second record, etc.)

       "OCK"
           an integer indicating the number of active keys in the record ( "1 <= OCK <= 2*ORDN" )

       "IT"
           an integer indicating the type of B*tree (1 for ".N01", 2 for ".N02")

       "IDX"
           an array of "ORDN" entries ("OCK" of which are active), each having the following
           format:

           "KEY"
               a fixed length character string of length ".LEx" ("LE1 =10", "LE2 = 30")

           "PUNT"
               a pointer to the ".N0x" record (if "PUNT > 0") or ".L0x" record (if "PUNT < 0")
               whose "IDX(1).KEY = KEY". "PUNT = 0" indicates an inactive entry. A positive
               "PUNT" indicates a branch to a hierarchically lower level index. The lowest level
               index ("PUNT < 0") points the leafs in the ".L0x" file.

   D. Format of ".L0x" files
       These files contain the full dictionary of searchable terms (".L01" for terms shorter than
       11 characters and ".L02" for terms longer than 10 characters). The ".L0x" file records
       have the following format (fields marked with "*" are 31-bit signed integers):

       "POS"*
           an integer indicating the relative record number (1 for the first record, 2 for the
           second record, etc.)

       "OCK"
           an integer indicating the number of active keys in the record ("1 < OCK <= 2*ORDF")

       "IT"
           an integer indicating the type of B*tree (1 for ".N01", 2 for ".N02")

       "PS"*
           is the immediate successor of "IDX[OCK].KEY" in this record (this is used to speed up
           sequential access to the file)

       "IDX"
           an array of "ORDN" entries ("OCK" of which are active), each having the following
           format:

           "KEY"
               a fixed length character string of length "LEx" ("LE1=10", "LE2=30")

           "INFO"
               a pointer to the ".IFP" record where the list of postings associated with "KEY"
               begins. This pointer consists of two 31-bit signed integers as follows:

               "INFO[1]"*
                       relative block number in ".IFP"

               "INFO[2]"*
                       offset (word number relative to 0) to postings list

   E. Format of ".IFP" file
       This file contains the list of postings for each dictionary term. Each list of postings
       has the format indicated below. The file is structured in blocks of 512 characters, where
       (for an initially loaded and compacted file) the lists of postings for each term are
       adjacent, except as noted below.

       The general format of each block is:

       "IFPBLK"
           a 31-bit signed integer indicating the Block number of this block (blocks are numbered
           from 1)

       "IFPREC"
           An array of 127 31-bit signed integers

       "IFPREC[1]" and "FPREC[2]" of the first block are a pointer to the next available position
       in the ".IFP" file.

       Pointers from ".L0x" to ".IFP" and pointers within ".IFP" consist of two 31-bit signed
       integers: the first integer is a block number, and the second integer is a word offset in
       "IFPREC" (e.g. the offset to the first word in "IFPREC" is 0). The list of postings
       associated with the first search term will therefore start at 1/0.

       Each list of postings consists of a header (5 double-words) followed by the actual list of
       postings (8 bytes for each posting). The header has the following format (each field is a
       31-bit signed integer):

       "IFPNXTB"*
           Pointer to next segment (Block number)

       "IFPNXTP"*
           Pointer to next segment (offset)

       "IFPTOTP"*
           Total number of postings (accurate only in first segment)

       "IFPSEGP"*
           Number of postings in this segment ("IFPSEGP <= IFPTOTP")

       "IFPSEGC"*
           Segment capacity (i.e. number of postings which can be stored in this segment)

       Each posting is a 64-bit string partitioned as follows:

       "PMFN"
           (24 bits) Master file number

       "PTAG"
           (16 bits) Field identifier (assigned from the "FST")

       "POCC"
           (8 bits) Occurrence number

       "PCNT"
           (16 bits) Term sequence number in field

       Each field is stored in a strict left-to-right sequence with leading zeros added if
       necessary to adjust the corresponding bit string to the right (this allows comparisons of
       two postings as character strings).

       The list of postings is stored in ascending "PMFN"/"PTAG"/"POCC"/"PCNT" sequence. When the
       inverted file is loaded sequentially (e.g. after a full inverted file generation with
       ISISINV), each list consists of one or more adjacent segments. If "IFPTOT <= 32768" then:
       "IFPNXTB/IFPNXTP = 0/0" and "IFPTOT = IFPSEGP = IFPSEGC".

       As updates are performed, additional segments may be created whenever new postings must be
       added. In this case a new segment with capacity "IFPTOTP" is created and linked to other
       segments (through the pointer "IFPNXTB"/"IFPNXTP") in such a way that the sequence
       "PMFN"/"PTAG"/"POCC"/"PCNT" is maintained. Whenever such a split occurs the postings of
       the segment where the new posting should have been inserted are equally distributed
       between this segment and the newly created segment.  New segments are always written at
       the end of the file (which is maintained in "IFPREC[1]"/"IFPREC[2]" of the first ".IFP"
       block.

       For example, assume that a new posting "Px" has to be inserted between "P2" and "P3" in
       the following list:

        +----------------------------+
        | 0 0 5 5 5 | P1 P2 P3 P4 P5 |
        +----------------------------+

       after the split (and assuming that the next available position in ".IFP" is 3/4) the list
       of postings will consist of the following two segments:

        +----------------------------+
        | 3 4 5 3 5 | P2 P2 Px -- -- |
        +--|-------------------------+
           |
        +--V-------------------------+
        | 0 0 5 3 5 | P3 P4 P5 -- -- |
        +----------------------------+

       In this situation, no new segment will be created until either segment becomes again full.

       As mentioned above, the posting lists are normally stored one after the other. However, in
       order to facilitate access to the ".IFP" file the segments are stored in such a way that:

       1.  the header and the first posting in each list (28 bytes) are never split between two
           blocks.

       2.  a posting is never split between two blocks; if there is not enough room in the
           current block the whole posting is stored in the next block.

LICENCE

       UNESCO has developed and owns the intellectual property of the CDS/ISIS software (in whole
       or in part, including all files and documentation, from here on referred to as CDS/ISIS)
       for the storage and retrieval of information.

       For complete text of licence visit <http://www.unesco.org/isis/files/winisislicense.html>.