NAME

       unidesc - Describe the contents of a Unicode text file

SYNOPSIS

       unidesc ([option flags]) (<file name>)

       If no input file name is supplied, unidesc reads from the standard input.

DESCRIPTION

       unidesc  describes the content of a Unicode text file by reporting the character ranges to
       which different portions of the text belong.  The ranges reported  include  both  official
       Unicode ranges and the constructed language ranges within the Private Use Areas registered
       with the Conscript Unicode Registry (http://www.evertype.com/standards/csur/).   For  each
       range  of  characters, unidesc prints the character or byte offset of the beginning of the
       range, the character or byte offset of the end of the range, and the name  of  the  range.
       Offsets start from 0.

       Since  the  ASCII  digits,  punctuation,  and whitespace characters are frequently used by
       other writing systems, by default these characters are treated as neutral, that is, as not
       belonging  exclusively to any particular character range.  These characters are treated as
       belonging to the range of whatever characters precede them.

       If the input begins with neutral characters, they are treated as belonging to the range of
       whatever  characters follow them. If the file consists entirely of neutral characters, the
       range is identified as Neutral followed by Basic Latin in square brackets.

       A magic number identifying the Unicode encoding is not part of the  Unicode  standard,  so
       pure  Unicode  files  do  not  contain a magic number.  However, informal conventions have
       arisen for this purpose.  If the command line flag -m is given, unidesc  will  attempt  to
       identify  the  Unicode subtype by examining the first few bytes of the input. If the input
       is identified as one of the two acceptable types, UTF-8 or native order  UTF-32,  it  will
       then  proceed to describe the contents of the input. Otherwise, it will report what it has
       learned and exit. Note that if the file does contain a magic number, you must use  the  -m
       flag.  Without  this flag unidesc assumes that the input consists of pure Unicode with the
       character data beginning immediately.  It will  therefore  be  thrown  off  by  the  magic
       number.

       By default, input is expected to be UTF-8. Native order UTF-32 is also acceptable.  UTF-32
       may be specified via the command line flag -u or, if the command line flag  -m  is  given,
       via the magic number.

COMMAND LINE FLAGS

       -b     Give file offsets in bytes rather than characters.

       -d     Treat the ASCII digits as belonging exclusively to the Basic Latin range.

       -h     Print usage information.

       -L     List the Unicode ranges alphabetically.

       -l     List the Unicode ranges by codepoint.

       -m     Check the file's magic number to determine the Unicode subtype.

       -p     Treat ASCII punctuation as belonging exclusively to the Basic Latin range.

       -r     Instead  of  listing  ranges as they are encountered, just list the ranges detected
              after all input has been read.

       -u     Input is native order UTF-32.

       -v     Print version information.

       -w     Treat ASCII whitespace as belonging exclusively to the Basic Latin range.

SEE ALSO

       uniname

REFERENCES

       Unicode Standard, version 5.0

AUTHOR

       Bill Poser
       billposer@alum.mit.edu

LICENSE

       GNU General Public License

                                            June, 2007                                 unidesc(1)