NAME

       unidesc - Describe the contents of a Unicode text file

SYNOPSIS

       unidesc ([option flags]) (<file name>)

       If no input file name is supplied, unidesc reads from the standard input.

DESCRIPTION

       unidesc describes the content of a Unicode text file by reporting the character ranges to which different
       portions of the text  belong.   The  ranges  reported  include  both  official  Unicode  ranges  and  the
       constructed  language  ranges within the Private Use Areas registered with the Conscript Unicode Registry
       (http://www.evertype.com/standards/csur/).  For each range of characters, unidesc prints the character or
       byte  offset of the beginning of the range, the character or byte offset of the end of the range, and the
       name of the range. Offsets start from 0.

       Since the ASCII digits, punctuation, and whitespace characters  are  frequently  used  by  other  writing
       systems, by default these characters are treated as neutral, that is, as not belonging exclusively to any
       particular character range.  These  characters  are  treated  as  belonging  to  the  range  of  whatever
       characters precede them.

       If  the  input  begins  with  neutral  characters, they are treated as belonging to the range of whatever
       characters follow them. If the file consists entirely of neutral characters, the range is  identified  as
       Neutral followed by Basic Latin in square brackets.

       A  magic  number  identifying  the  Unicode encoding is not part of the Unicode standard, so pure Unicode
       files do not contain a magic number.  However, informal conventions have arisen for this purpose.  If the
       command  line  flag  -m  is  given, unidesc will attempt to identify the Unicode subtype by examining the
       first few bytes of the input. If the input is identified as one of the two  acceptable  types,  UTF-8  or
       native  order  UTF-32,  it  will  then  proceed to describe the contents of the input. Otherwise, it will
       report what it has learned and exit. Note that if the file does contain a magic number, you must use  the
       -m  flag.  Without  this  flag unidesc assumes that the input consists of pure Unicode with the character
       data beginning immediately.  It will therefore be thrown off by the magic number.

       By default, input is expected to be UTF-8. Native  order  UTF-32  is  also  acceptable.   UTF-32  may  be
       specified via the command line flag -u or, if the command line flag -m is given, via the magic number.

COMMAND LINE FLAGS

       -b     Give file offsets in bytes rather than characters.

       -d     Treat the ASCII digits as belonging exclusively to the Basic Latin range.

       -h     Print usage information.

       -L     List the Unicode ranges alphabetically.

       -l     List the Unicode ranges by codepoint.

       -m     Check the file's magic number to determine the Unicode subtype.

       -p     Treat ASCII punctuation as belonging exclusively to the Basic Latin range.

       -r     Instead  of  listing ranges as they are encountered, just list the ranges detected after all input
              has been read.

       -u     Input is native order UTF-32.

       -v     Print version information.

       -w     Treat ASCII whitespace as belonging exclusively to the Basic Latin range.

SEE ALSO

       uniname

REFERENCES

       Unicode Standard, version 5.0

AUTHOR

       Bill Poser
       billposer@alum.mit.edu

LICENSE

       GNU General Public License

                                                   June, 2007                                         unidesc(1)