NAME

       ids2ngram - generate n-gram data file from ids file

SYNOPSIS

       ids2ngram [option]... ids_file...

DESCRIPTION

       ids2ngram generates idngram file, which is a sorted [id1,..,idN,freq] array, from binary id stream files.
       Here, the id stream files are always generated by mmseg or slmseg. Basically, it finds all occurrence of
       n-words tuples (i.e. the tuple of (id1,..,idN)), and sorts these tuples by the lexicographic order of the
       ids make up the tuples, then write them to specified output file.

INPUT

       The input file is presented as a binary id stream, which looks like:
           [id0,...,idX]

OPTIONS

       All the following options are mandatory.

       -n,--NMax N
           Generates N-gram result. ids2ngram does only support uni-gram, bi-gram, and trigram, so any number
           not in the range of 1..3 is not valid.

       -s,--swap swap-file
           Specify the temporary intermediate file.

       -o, --out output-file
           Specify the result idngram file, e.g. the array of [id1, ..., idN, freq]

       -p, --para N
           Specify the maximum n-gram items per paragraph. ids2ngram writes to the temporary file on a per-
           paragraph basis. Every time it writes a paragraph out, it frees the corresponding memory allocated
           for it. When your computer system permits, a higher N is suggested. This can speed up the processing
           speed because of less I/O.

EXAMPLE

       Following example will use three input idstream file idsfile[1,2,3] to generate the idngram file
       all.id3gram. Each para (internal map size or hash size) would be 1024000, using swap file for temp
       result. All temp para result would eventually be merged to got the final result.

       ids2ngram -n 3 -s /tmp/swap -o all.id3gram -p 1024000 idsfile1 idsfile2 idsfile3

AUTHOR

       Originally written by Phill.Zhang <phill.zhang@sun.com>.  Currently maintained by Kov.Chai
       <tchaikov@gmail.com>.

SEE ALSO

       mmseg(1), slmseg(1), slmbuild (1).