Ubuntu Manpage: po4a-gettextize - convert an original file (and its translation) to a PO file

NAME

       po4a-gettextize - convert an original file (and its translation) to a PO file

SYNOPSIS

       po4a-gettextize -f fmt -m master.doc -l XX.doc -p XX.po

       (XX.po is the output, all others are inputs)

DESCRIPTION

       po4a (PO for anything) eases the maintenance of documentation translation using the
       classical gettext tools. The main feature of po4a is that it decouples the translation of
       content from its document structure.  Please refer to the page po4a(7) for a gentle
       introduction to this project.

       The po4a-gettextize script helps you converting your previously existing translations into
       a po4a-based workflow. This is only to be done once to salvage an existing translation
       while converting to po4a, not on a regular basis after the conversion of your project.
       This tedious process is explained in details in Section 'Converting a manual translation
       to po4a' below.

       You must provide both a master file (e.g., the source in English) and an existing
       translated file (e.g., a previous translation attempt without po4a). If you provide more
       than one master or translation files, they will be used in sequence, but it may be easier
       to gettextize each page or chapter separately and then use msgmerge to merge all produced
       PO files. As you wish.

       If the master document has non-ASCII characters, the new generated PO file will be in
       UTF-8. If the master document is completely in ASCII, the generated PO will use the
       encoding of the translated input document.

OPTIONS

-f, --format
Format of the documentation you want to handle. Use the --help-format option to see
the list of available formats.

-m, --master
File containing the master document to translate. You can use this option multiple
times if you want to gettextize multiple documents.

-M, --master-charset
Charset of the file containing the document to translate.

-l, --localized
File containing the localized (translated) document. If you provided multiple master
files, you may wish to provide multiple localized file by using this option more than
once.

-L, --localized-charset
Charset of the file containing the localized document.

-p, --po
File where the message catalog should be written. If not given, the message catalog
will be written to the standard output.

-o, --option
Extra option(s) to pass to the format plugin. See the documentation of each plugin for
more information about the valid options and their meanings. For example, you could
pass '-o tablecells' to the AsciiDoc parser, while the text parser would accept '-o
tabs=split'.

-h, --help
Show a short help message.

--help-format
List the documentation formats understood by po4a.

-k --keep-temps
Keep the temporary master and localized POT files built before merging. This can be
useful to understand why these files get desynchronized, leading to gettextization
problems.

-V, --version
Display the version of the script and exit.

-v, --verbose
Increase the verbosity of the program.

-d, --debug
Output some debugging information.

--msgid-bugs-address email@address
Set the report address for msgid bugs. By default, the created POT files have no
Report-Msgid-Bugs-To fields.

--copyright-holder string
Set the copyright holder in the POT header. The default value is "Free Software
Foundation, Inc."

--package-name string
Set the package name for the POT header. The default is "PACKAGE".

--package-version string
Set the package version for the POT header. The default is "VERSION".

Converting a manual translation to po4a
po4a-gettextize synchronizes the master and localized files to extract their content into
a PO file. The content of the master file gives the msgid while the content of the
localized file gives the msgstr. This process is somewhat fragile: the Nth string of the
translated file is supposed to be the translation of the Nth string in the original.

Gettextization works best if you manage to retrieve the exact version of the original
document that was used for translation. Even so, you may need to fiddle with both master
and localized files to align their structure if it was changed by the original translator,
so working on files' copies is advised.

Internally, each po4a parser reports the syntactical type of each extracted strings. This
is how desynchronization are detected during the gettextization. In the example depicted
below, it is very unlikely that the 4th string in translation (of type 'chapter') is the
translation of the 4th string in original (of type 'paragraph'). It is more likely that a
new paragraph was added to the original, or that two original paragraphs were merged
together in the translation.

Original Translation

chapter chapter
paragraph paragraph
paragraph paragraph
paragraph chapter
chapter paragraph
paragraph paragraph

po4a-gettextize will verbosely diagnose any structure desynchronization. When this
happens, you should manually edit the files to add fake paragraphs or remove some content
here and there until the structure of both files actually match. Some tricks are given
below to salvage the most of the existing translation while doing so.

If you are lucky enough to have a perfect match in the file structures out of the box,
building a correct PO file is a matter of seconds. Otherwise, you will soon understand why
this process has such an ugly name :) Even so, gettextization often remains faster than
translating everything again. I gettextized the French translation of the whole Perl
documentation in one day despite the many synchronization issues. Given the amount of text
(2MB of original text), restarting the translation without first salvaging the old
translations would have required several months of work. In addition, this grunt work is
the price to pay to get the comfort of po4a. Once converted, the synchronization between
master documents and translations will always be fully automatic.

After a successful gettextization, the produced documents should be manually checked for
undetected disparities and silent errors, as explained below.

Hints and tricks for the gettextization process

The gettextization stops as soon as a desynchronization is detected. When this happens,
you need to edit the files as much as needed to re-align the files' structures.
po4a-gettextize is rather verbose when things go wrong. It reports the strings that don't
match, their positions in the text, and the type of each of them. Moreover, the PO file
generated so far is dumped as gettextization.failed.po for further inspection.

Here are some tricks to help you in this tedious process and ensure that you salvage the
most of the previous translation:

• Remove all extra content of the translations, such as the section giving credits to
the translators. They should be added separately to po4a as addenda (see po4a(7)).

• When editing the files to align their structures, prefer editing the translation if
possible. Indeed, if the changes to the original are too intrusive, the old and new
versions will not be matched during the first po4a run after gettextization (see
below). Any unmatched translation will be dumped anyway. That being said, you still
want to edit the original document if it's too hard to get the gettextization to
proceed otherwise, even if it means that one paragraph of the translation is dumped.
The important thing is to get a first PO file to start with.

• Do not hesitate to kill any original content that would not exist in the translated
version. This content will be automatically reintroduced afterward, when synchronizing
the PO file with the document.

• You should probably inform the original author of any structural change in the
translation that seems justified. Issues in the original document should reported to
the author. Fixing them in your translation only fixes them for a part of the
community. Plus, it is impossible to do so when using po4a ;) But you probably want to
wait until the end of the conversion to po4a before changing the original files.

• Sometimes, the paragraph content does match, but not their types. Fixing it is rather
format-dependent. In POD and man, it often comes from the fact that one of them
contains a line beginning with a white space while the other does not. In those
formats, such paragraph cannot be wrapped and thus become a different type. Just
remove the space and you are fine. It may also be a typo in the tag name in XML.

Likewise, two paragraphs may get merged together in POD when the separating line
contains some spaces, or when there is no empty line between the =item line and the
content of the item.

• Sometimes, the desynchronization message seems odd because the translation is attached
to the wrong original paragraph. It is the sign of an undetected issue earlier in the
process. Search for the actual desynchronization point by inspecting the file
gettextization.failed.po that was produced, and fix the problem where it really is.

• Other issues may come from duplicated strings in either the original or translation.
Duplicated strings are merged in PO files, with two references. This constitutes a
difficulty for the gettextization algorithm, that is a simple one to one pairing
between the msgids of both the master and the localized files. It is however believed
that recent versions of po4a deal properly with duplicated strings, so you should
report any remaining issue that you may encounter.

Reviewing files produced by po4a-gettextize
Any file produced by po4a-gettextize should be manually reviewed, even when the script
terminates successfully. You should skim over the PO file, ensuring that the msgid and
msgstr actually match. It is not necessary to ensure that the translation is perfectly
correct yet, as all entries are marked as fuzzy translations anyway. You only need to
check for obvious matching issues because badly matched translations will be dumped in
subsequent steps while you want to salvage them.

Fortunately, this step does not require to master the target languages as you only want to
recognize similar elements in each msgid and its corresponding msgstr. As a speaker of
French, English, and some German myself, I can do this for all European languages at
least, even if I cannot say one word of most of these languages. I sometimes manage to
detect matching issues in non-Latin languages by looking at string length, phrase
structures (does the amount of interrogation marks match?) and other clues, but I prefer
when someone else can review those languages.

If you detect a mismatch, edit the original and translation files as if po4a-gettextize
reported an error, and try again. Once you have a decent PO file for your previous
translation, backup it until you get po4a working correctly.

Running po4a for the first time
The easiest way to setup po4a is to write a po4a.conf configuration file, and use the
integrated po4a program (po4a-updatepo and po4a-translate are deprecated). Please check
the "CONFIGURATION FILE" Section in po4a(1) documentation for more details.

When po4a runs for the first time, the current version of the master documents will be
used to update the PO files containing the old translations that you salvaged through
gettextization. This can take quite a long time, because many of the msgids of from the
gettextization do not exactly match the elements of the POT file built from the recent
master files. This forces gettext to search for the closest one using a costly string
proximity algorithm. For example, the first run over the Perl documentation's French
translation (5.5 MB PO file) took about 48 hours (yes, two days) while the subsequent ones
only take seconds.

Moving your translations to production
After this first run, the PO files are ready to be reviewed by translators. All entries
were marked as fuzzy in the PO file by po4a-gettextization, forcing their careful review
before use. Translators should take each entry to verify that the salvaged translation
actually match the current original text, update the translation on need, and remove the
fuzzy markers.

Once enough fuzzy markers are removed, po4a will start generating the translation files on
disk, and you're ready to move your translation workflow to production. Some projects find
it useful to rely on weblate to coordinate between translators and maintainers, but that's
beyond po4a' scope.

AUTHORS

        Denis Barbier <barbier@linuxfr.org>
        Nicolas François <nicolas.francois@centraliens.net>
        Martin Quinson (mquinson#debian.org)

COPYRIGHT AND LICENSE