NAME

       Scalar::String - string aspects of scalars

SYNOPSIS

               use Scalar::String
                       qw(sclstr_is_upgraded sclstr_is_downgraded);

               if(sclstr_is_upgraded($value)) { ...
               if(sclstr_is_downgraded($value)) { ...

               use Scalar::String qw(
                       sclstr_upgrade_inplace sclstr_upgraded
                       sclstr_downgrade_inplace sclstr_downgraded
               );

               sclstr_upgrade_inplace($value);
               $value = sclstr_upgraded($value);
               sclstr_downgrade_inplace($value);
               $value = sclstr_downgraded($value);

DESCRIPTION

       This module is about the string part of plain Perl scalars.  A scalar has a string value,
       which is notionally a sequence of Unicode codepoints, but may be internally encoded in
       either ISO-8859-1 or UTF-8.  In places, and more so in older versions of Perl, the
       internal encoding shows through.  To fully understand Perl strings it is necessary to
       understand these implementation details.

       This module provides functions to classify a string by encoding and to encode a string in
       a desired way.

       This module is implemented in XS, with a pure Perl backup version for systems that can't
       handle XS.

STRING ENCODING

       ISO-8859-1 is a simple 8-bit character encoding, which represents the first 256 Unicode
       characters (codepoints 0x00 to 0xff) in one octet each.  This is how strings were
       historically represented in Perl.  A string represented this way is referred to as
       "downgraded".

       UTF-8 is a variable-width character encoding, which represents all possible Unicode
       codepoints in differing numbers of octets.  A design feature of UTF-8 is that ASCII
       characters (codepoints 0x00 to 0x7f) are each represented in a single octet, identically
       to their ISO-8859-1 encoding.  Perl has its own variant of UTF-8, which can handle a wider
       range of codepoints than Unicode formally allows.  A string represented in this variant
       UTF-8 is referred to as "upgraded".

       A Perl string is physically represented as a string of octets along with a flag that says
       whether the string is downgraded or upgraded.  At this level, to determine the Unicode
       codepoints that are represented requires examining both parts of the representation.  If
       the string contains only ASCII characters then the octet sequence is identical in either
       encoding, but Perl still maintains an encoding flag on such a string.  A string is always
       either downgraded or upgraded; it is never both or neither.

       When handling string input, it is good form to operate only on the Unicode characters
       represented by the string, ignoring the manner in which they are encoded.  Basic string
       operations such as concatenation work this way (except for a bug in perl 5.6.0), so simple
       code written in pure Perl is generally safe on this front.  Pieces of character-based code
       can pass around strings among themselves, and always get consistent behaviour, without
       worrying about the way in which the characters are encoded.

       However, due to an historical accident, a lot of C code that interfaces with Perl looks at
       the octets used to represent a string without also examining the encoding flag.  Such code
       gives inconsistent behaviour for the same character sequence represented in the different
       ways.  In perl 5.6, many pure Perl operations (such as regular expression matching) also
       work this way, though some of them can be induced to work correctly by using the utf8
       pragma.  In perl 5.8, regular expression matching is character-based by default, but many
       I/O functions (such as "open") are still octet-based.

       Where code that operates on the octets of a string must be used by code that operates on
       characters, the latter needs to pay attention to the encoding of its strings.  Commonly,
       the octet-based code expects its input to be represented in a particular encoding, in
       which case the character-based code must oblige by forcing strings to that encoding before
       they are passed in.  There are other usage patterns too.

       You will be least confused if you think about a Perl string as a character sequence plus
       an encoding flag.  You should normally operate on the character sequence and not care
       about the encoding flag.  Occasionally you must pay attention to the flag in addition to
       the characters.  Unless you are writing C code, you should try not to think about a string
       the other way round, as an octet sequence plus encoding flag.

FUNCTIONS

       Each "sclstr_" function takes one or more scalar string arguments to operate on.  These
       arguments must be strings; giving non-string arguments will cause mayhem.  See "is_string"
       in Params::Classify for a way to check for stringness.  Only the string value of the
       scalar is used; the numeric value is completely ignored, so dualvars are not a problem.

   Classification
       sclstr_is_upgraded(VALUE)
           Returns a truth value indicating whether the provided string VALUE is in upgraded
           form.

       sclstr_is_downgraded(VALUE)
           Returns a truth value indicating whether the provided string VALUE is in downgraded
           form.

   Regrading
       sclstr_upgrade_inplace(VALUE)
           Modifies the string VALUE in-place, so that it is in upgraded form, regardless of how
           it was encoded before.  The character sequence that it represents is unchanged.

           A cleaner interface to this operation is the non-mutating "sclstr_upgraded".

       sclstr_upgraded(VALUE)
           Returns a string that represents the same character sequence as the string VALUE, and
           is in upgraded form (regardless of how VALUE is encoded).

       sclstr_downgrade_inplace(VALUE[, FAIL_OK])
           Modifies the string VALUE in-place, so that it is in downgraded form, regardless of
           how it was encoded before.  The character sequence that it represents is unchanged.
           If the string cannot be downgraded, because it contains a non-ISO-8859-1 character,
           then by default the function "die"s, but if FAIL_OK is present and true then it will
           return leaving VALUE unmodified.

           A cleaner interface to this operation is the non-mutating "sclstr_downgraded".

       sclstr_downgraded(VALUE[, FAIL_OK])
           Returns a string that represents the same character sequence as the string VALUE, and
           is in downgraded form (regardless of how VALUE is encoded).  If the string cannot be
           represented in downgraded form, because it contains a non-ISO-8859-1 character, then
           by default the function "die"s, but if FAIL_OK is present and true then it will return
           VALUE in its original upgraded form.

SEE ALSO

       utf8

AUTHOR

       Andrew Main (Zefram) <zefram@fysh.org>

COPYRIGHT

       Copyright (C) 2009, 2010, 2011 Andrew Main (Zefram) <zefram@fysh.org>

LICENSE

       This module is free software; you can redistribute it and/or modify it under the same
       terms as Perl itself.