Warning, /sdk/pology/doc/user/misctools.docbook is written in an unsupported language. File is not indexed.

 <?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
  "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd">
 
 <chapter id="ch-misctools">
 <title>Miscellaneous Tools</title>
 
 <para>This chapter describes various smaller standalone tools in Pology, which do not introduce any major PO processing concepts nor can be grouped under a common topic.</para>
 
 <!-- ======================================== -->
 <sect1 id="sec-mirewrap">
 <title>Rewrapping PO Files with <command>porewrap</command></title>
 
 <para>The <command>porewrap</command> script does one simple thing: it rewraps message strings (<varname>msgid</varname>, <varname>msgstr</varname>, etc.) in PO files. Gettext's tools, e.g. <command>msgcat</command>, can be used for rewrapping as well, so what is the reason of existence of <command>porewrap</command>? The lesser reason is convenience. Arbitrary number of PO file paths can be given to it as arguments, as well as directory paths which will be recursively search for PO files. The more important reason is that Pology can also perform "fine" wrapping, as described in <xref linkend="sec-cmwrap"/>. Thus, running:
 <programlisting language="bash">
 $ porewrap --no-wrap --fine-wrap somedir/
 </programlisting>
 will rewrap all PO files found in <filename>somedir/</filename> and below, such that basic wrapping (on column) is disabled (<literal>--no-wrap</literal>), while fine wrapping (on logical breaks) is enabled (<literal>--fine-wrap</literal>).</para>
 
 <para>Other than from command line options, <command>porewrap</command> will also consult the PO file header and the user configuration, for the wrapping mode. Command line options have the highest priority, followed by the PO header, and the user configuration at the end. For details on how to set the wrapping mode in PO headers, see the description of <literal>X-Wrapping</literal> header field in <xref linkend="sec-cmheader"/>. If none of these sources specify the wrapping mode, <command>porewrap</command> will apply basic wrapping.</para>
 
 <sect2 id="sec-mirwopts">
 <title>Command Line Options</title>
 
 <para>Options specific to <command>porewrap</command>:
 <variablelist>
 
 <varlistentry>
 <term><option>-v</option>, <option>--verbose</option></term>
 <listitem>
 <para>Since <command>porewrap</command> just opens and writes back all the PO files given to it, it normally does not report anything. But this option can be issued for it to report PO file paths as they have been written out.</para>
 </listitem>
 </varlistentry>
 
 </variablelist>
 </para>
 
 <para>
 Options common with other Pology tools:
 <variablelist>
 
 <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
             href="stdopt-wrapping.docbook"/>
 
 <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
             href="stdopt-filesfrom.docbook"/>
 
 </variablelist>
 </para>
 
 </sect2>
 
 <sect2 id="sec-mirwcfg">
 <title>User Configuration</title>
 
 <para><command>porewrap</command> reads the wrapping mode fields as described in <xref linkend="sec-cmwrcfg"/>, from its <literal>[porewrap]</literal> section.</para>
 
 </sect2>
 
 </sect1>
 
 <!-- ======================================== -->
 <sect1 id="sec-miselfmerge">
 <title>Self-Merging PO Files with <command>poselfmerge</command></title>
 
 <para>Normally, PO files are periodically merged with latest PO templates, to introduce changes from the source material while preserving as much of the existing translation as possible. <command>poselfmerge</command>, on the other hand, will merge the PO file with "itself". More precisely, it will derive the temporary template version of the PO file (by cleaning it from translations and other details), and then merge the original PO file with the derived template, by calling <command>msgmerge</command> internally. This can have several uses:
 <itemizedlist>
 
 <listitem>
 <para>The fuzzy matching algorithm of <command>msgmerge</command> is extremely fast and robust, but treats all messages the same and in isolation, without trying out more complicated (and necessarily much slower) heuristic criteria. This can cause the translator to spend more time updating a fuzzy message than it would take to translate it from scratch. <command>poselfmerge</command> can be therefore instructed to go over all fuzzy messages created by merging, and apply additional heuristics to determine whether to leave the message fuzzy or to clean it up and make it fully untranslated.</para>
 </listitem>
 
 <listitem>
 <para>Sometimes the PO file can contain a number of quite similar longer messages (this is especially the case when <link linkend="ch-summit">translating in summit</link>). A capable PO editor should automatically offer the previous translation on the next similar message (by using internal translation memory), and show the what the small differences in the original text are, thus greately speeding up the translation of that message. If, however, the PO editor is not that capable, or you use a plain text editor, while translating you can simply skip every long message that looks familiar, and afterwards run <command>poselfmerge</command> on the PO file to introduce fuzzy matches on those messages.</para>
 </listitem>
 
 <listitem>
 <para>More generally, if your PO editor does not have (a good enough) translation memory feature, or you edit PO files with a plain text editor, you can instruct <command>poselfmerge</command> to use one or more <emphasis>PO compendia</emphasis> to provide additional exact and fuzzy matches. This is essentially the batch application of translation memory. <xref linkend="sec-cbcompend"/> provides some hints on how to create and maintain PO compendia.</para>
 </listitem>
 
 </itemizedlist>
 </para>
 
 <para>Arguments to <command>poselfmerge</command> are any number of PO file paths or directories to search for PO files, which will be modified in place:
 <programlisting language="bash">
 $ poselfmerge foo.po bar.po somedir/
 </programlisting>
 However, this run will do almost nothing (except possibly rewrap files), just as <command>msgmerge</command> would do nothing if the same template were used twice. Instead, all special processing must be requested by command line options, or activated through the <link linkend="sec-cmconfig">user configuration</link> to avoid issuing some options with same values all the time.</para>
 
 <sect2 id="sec-mismopts">
 <title>Command Line Options</title>
 
 <para>Options specific to <command>poselfmerge</command>:
 <variablelist>
 
 <varlistentry>
 <term><option>-A <replaceable>RATIO</replaceable></option>, <option>--min-adjsim-fuzzy=<replaceable>RATIO</replaceable></option></term>
 <listitem>
 <para>The minimum required "adjust similarity" between the old and the new orginal text in a fuzzy message, in order to accept it and not clean it to untranslated state. The similarity is expressed as the ratio in range 0.0-1.0, with 0.0 meaning no similarity and 1.0 no difference. A practical range is 0.6-0.8. If this option is not issued, fuzzy messages are kept as they are (as if 0.0 would be given).</para>
 
 <para>The requirement for computation of adjusted similarity is that fuzzy messages contain previous strings, i.e. that the PO file was originally merged with <option>--previous</option> to <command>msgmerge</command>.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-b</option>, <option>--rebase-fuzzies</option></term>
 <listitem>
 <para>Normally, when merging with template, the untranslated and fuzzy messages already present in the PO file are not checked again for approximate matches. This is on the one hand side a performance measure (why fuzzy match again something that was already matched before?), and on the other hand a safety measure (higher trust in an old fuzzy match based on the PO file itself than e.g. a new match from an arbitrary compendium). By issuing this option, prior to merging all untranslated messages are removed from the PO file, and all fuzzy messages which have previous strings are converted to obsolete previous messages. This activates fuzzy matching on untranslated messages (e.g. if new compendium given, or for similar messages skipped during translation), and puts possibly better previous strings on fuzzy messages (unless an exact match is found in compendium).</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-C <replaceable>POFILE</replaceable></option>, <option>--compendium=<replaceable>POFILE</replaceable></option></term>
 <listitem>
 <para>The PO file to use as compendium on merging, to produce more exact and fuzzy matches. This option can be repeated to add several compendia.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-v</option>, <option>--verbose</option></term>
 <listitem>
 <para><command>poselfmerge</command> normally operates silently, and this option requests some progress information. Quite useful if processing a large collection of PO files, because merging and post-merge processing can take <emphasis>a lot</emphasis> of time (especially in presence of compendium).</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-W <replaceable>NUMBER</replaceable></option>, <option>--min-words-exact=<replaceable>NUMBER</replaceable></option></term>
 <listitem>
 <para>When an exact match for an untranslated message is produced from the compendium, it is not always safe to silently accept it, because the compendium may contain translations from contexts totally unrelated with the current PO file. The shorter the message, the higher the chance that translation will not be suitable in current context. This option provides the minimum number of words (in the original) to accept an exact match from the compendium, or else the message is made fuzzy. The reasonable value depends on the relation between the source and the target language, with 5 to 10 probably being on the safe side.</para>
 
 <para>Note that afterwards you can see when an exact match has been demoted into a fuzzy one, by that message not having previous strings (<literal>#| msgid "..."</literal>, etc.).</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-x</option>, <option>--fuzzy-exact</option></term>
 <listitem>
 <para>This option is used to unconditionally demote exact matches from the compendium into fuzzy messages (e.g. regardless of the length of the text, as done by <option>-W</option>/<option>--min-words-exact</option>). This may be needed, for example, when there is a strict review procedure in place, and the compendium is built from unreviewed translations.</para>
 </listitem>
 </varlistentry>
 
 </variablelist>
 </para>
 
 <para>
 Options common with other Pology tools:
 <variablelist>
 
 <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
             href="stdopt-wrapping.docbook"/>
 
 <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
             href="stdopt-filesfrom.docbook"/>
 
 </variablelist>
 </para>
 
 </sect2>
 
 <sect2 id="sec-mismcfg">
 <title>User Configuration</title>
 
 <para>It is likely that the translator will have a certain personal preference of the various match acceptance criteria provided by command line options. Instead of issuing those options all the time, the following user configuration fields may be set:
 <variablelist>
 
 <varlistentry>
 <term><literal>[poselfmerge]/fuzzy-exact=[yes|*no]</literal></term>
 <listitem>
 <para>Counterpart to the <option>-x</option>/<option>--fuzzy-exact</option> option.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><literal>[poselfmerge]/min-adjsim-fuzzy</literal></term>
 <listitem>
 <para>Counterpart to the <option>-A</option>/<option>--min-adjsim-fuzzy</option> option.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><literal>[poselfmerge]/min-words-exact</literal></term>
 <listitem>
 <para>Counterpart to the <option>-W</option>/<option>--min-words-exact</option> option.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><literal>[poselfmerge]/rebase-fuzzies=[yes|*no]</literal></term>
 <listitem>
 <para>Counterpart to the <option>-b</option>/<option>--rebase-fuzzies</option> option.</para>
 </listitem>
 </varlistentry>
 
 </variablelist>
 Of course, command line options can be issued to override the user configuration fields when necessary.</para>
 
 <para><command>poselfmerge</command> also reads the wrapping mode fields as described in <xref linkend="sec-cmwrcfg"/>, from its <literal>[poselfmerge]</literal> section.</para>
 
 </sect2>
 
 </sect1>
 
 <!-- ======================================== -->
 <sect1 id="sec-mimtrans">
 <title>Machine Translation with <command>pomtrans</command></title>
 
 <para><emphasis>Machine translation</emphasis> is the process where a computer program is used to produce translation of more than a trivial piece of text, starting from single sentences, over paragraphs, to full documents. There are debates on how useful machine translation is right now and how much better it could become in the future, and there is a steady line of research in that direction. Limiting to widely available examples of machine translation software today, it is safe to say that, on the one hand, machine translation can preserve a lot of the meaning of the original and thus be very useful to the reader who needs to grasp the main points of the text, but on the other hand, are not at all passable for producing translations of the quality expected of human translators who are native speaker of the target language.</para>
 
 <para>As far as Pology is concerned, the question of machine translation reduces to this: would it increase the efficiency of translation if PO files were first machine-translated, and then manually corrected by a human translator? There is no general answer to this question, as it depends stronly on all elements in the chain: the quality of machine translation software, the source language, the target language, and the human translator. Be that as it may, Pology provides the <command>pomtrans</command> script, which can fill in untranslated messages in PO files by passing original text through various machine translation services.</para>
 
 <para><command>pomtrans</command> has two principal modes of operation. The more straightforward is the direct mode, where original texts are simply <varname>msgid</varname> strings in the given PO file. In this mode, PO files can be machine-translated with:
 <programlisting language="bash">
 $ pomtrans <replaceable>transerv</replaceable> -t <replaceable>lang</replaceable> <replaceable>paths...</replaceable>
 </programlisting>
 The first argument is the translation service keyword, chosen from one known to <command>pomtrans</command>. The <option>-t</option> option specifies the target language; it may not be necessary if processed PO files have the <literal>Language:</literal> header field properly set. The source language is assumed to be English, but there is an option to specify another source language. Afterwards an arbitrary number of paths follow, which may be either single PO files or directories which will be recursively searched for PO files.</para>
 
 <para><command>pomtrans</command> will try to translate only untranslated messages, and not fuzzy messages. When it translates a message, by default it will make it fuzzy as well, meaning that a human should go through all machine-translated messages. These defaults are based on the perceived current quality of most machine translation services. There are several command line options to change this behavior.</para>
 
 <para>The other mode of operation is the parallel mode. Here <command>pomtrans</command> takes the original text to be the translation into another language, i.e. <varname>msgstr</varname> strings from a PO file translated into another language. For example, if a PO file should be translated into Spanish (i.e. from English to Spanish), and that same PO file is available fully translated into French (i.e. from English to French), then <command>pomtrans</command> could be used to translate from French to Spanish. This is done in the following way:
 <programlisting language="bash">
 $ pomtrans <replaceable>transerv</replaceable> -s <replaceable>lang1</replaceable> -t <replaceable>lang2</replaceable> -p <replaceable>search</replaceable>:<replaceable>replace</replaceable> <replaceable>paths...</replaceable>
 </programlisting>
 As in direct mode, the first argument is the translation service. Then both the source (<option>-s</option>) and the target language (<option>-t</option>) are specified; again, if PO files have their <literal>Language:</literal> header fields set, these options are not necessary. The perculiar here is the <option>-p</option> option, which specifies two strings, separated by colon. These are used to construct paths to source language PO files, by replacing the first string in paths of target language PO files with the second string. For example, if the file tree is:
 <programlisting>
 foo/
     po/
         alpha/
             alpha.pot
             fr.po
             es.po
         bravo/
             bravo.pot
             fr.po
             es.po
 </programlisting>
 then the invocation could be:
 <programlisting language="bash">
 $ cd .../foo/
 $ pomtrans <replaceable>transerv</replaceable> -s fr -t es -p es.:fr. po/*/es.po
 </programlisting>
 In case a PO file in target language does not have a counterpart in source language, it is simply skipped.</para>
 
 <para>There is another variation of the parallel mode, where source language texts are drawn not from counterpart PO files, but from a single, compendium PO file in source language. This mode is engaged by giving the path to that compendium with the <option>-c</option> option, instead of the <option>-p</option> option for path replacement.</para>
 
 <sect2 id="sec-mimtopts">
 <title>Command Line Options</title>
 
 <para>Options specific to <command>pomtrans</command>:
 <variablelist>
 
 <varlistentry>
 <term><option>-a <replaceable>CHARS</replaceable></option>, <option>--accelerator=<replaceable>CHARS</replaceable></option></term>
 <listitem>
 <para>Characters used as <link linkend="sec-poaccel">accelerator markers</link> in user interface messages. They should be removed from the source language text before translation, in order not to confuse the translation service.<footnote>
 <para>This also means that, at the moment, machine-translated text has no accelerator when the original text did have one. Some heuristics may be implemented in the future to add the accelerator to translated text as well.</para>
 </footnote></para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-c <replaceable>FILE</replaceable></option>, <option>--parallel-compendium=<replaceable>FILE</replaceable></option></term>
 <listitem>
 <para>The path to source language compendium, in parallel translation mode.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-l</option>, <option>--list-transervs</option></term>
 <listitem>
 <para>Lists known translation services (the keywords which can be the first argument to <command>pomtrans</command>).</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-m</option>, <option>--flag-mtrans</option></term>
 <listitem>
 <para>Adds the <literal>mtrans</literal> flag to each machine-translated message. This may be useful to positively identify machine-translated messages in the resulting PO file, as otherwise they are simply fuzzy.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-M <replaceable>MODE</replaceable></option>, <option>--translation-mode=<replaceable>MODE</replaceable></option></term>
 <listitem>
 <para>Translation services need as input the mode in which to operate, usually the source and target language at minimum. By default the translation mode is constructed based on source and target languages, but this is sometimes not precise enough. This option can be used to issue a custom mode string for the chosen translation service, overriding the default construction. The format of the mode string is translation service dependent, check documentation of respective translation services for details.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-n</option>, <option>--no-fuzzy-flag</option></term>
 <listitem>
 <para>By default machine-translated messages are made fuzzy, which is prevented by this option. It goes without saying that this is dangerous at current state of the art in machine translation, and should be used only in very specific scenarios (e.g. high quality machine translation between two dialects of the same language).</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-p <replaceable>SEARCH</replaceable>:<replaceable>REPLACE</replaceable></option>, <option>--parallel-catalogs=<replaceable>SEARCH</replaceable>:<replaceable>REPLACE</replaceable></option></term>
 <listitem>
 <para>The string to search for in paths of target language PO files, and the string to replace them with to construct paths of source language PO files, in parallel translation mode.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-s <replaceable>LANG</replaceable></option>, <option>--source-lang=<replaceable>LANG</replaceable></option></term>
 <listitem>
 <para>The source language code, i.e. the language which is being translated from.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-t <replaceable>LANG</replaceable></option>, <option>--target-lang=<replaceable>LANG</replaceable></option></term>
 <listitem>
 <para>The target language code, i.e. the language which is being translated into.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-T <replaceable>PATH</replaceable></option>, <option>--transerv-bin=<replaceable>PATH</replaceable></option></term>
 <listitem>
 <para>If the selected translation service is (or can be) a program on the local computer, this option can be used to specify the path to its executable file, if it is not in the <envar>PATH</envar>.</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term><option>-d <replaceable>DIRECTORY</replaceable></option>, <option>--data-directory=<replaceable>DIRECTORY</replaceable></option></term>
 <listitem>
 <para>If the selected translation service can use a local directory of translation data, this option can be used to specify the path to that directory. It is equivalent to Apertium’s <option>-d</option> parameter.</para>
 </listitem>
 </varlistentry>
 
 </variablelist>
 </para>
 
 </sect2>
 
 <sect2 id="sec-mimtservs">
 <title>Supported Machine Translation Services</title>
 
 <para>Currently supported translation services are as follows (with keyword in parenthesis):
 <variablelist>
 
 <varlistentry>
 <term>Apertium (<literal>apertium</literal>)</term>
 <listitem>
 <para><ulink url="http://www.apertium.org/">Apertium</ulink> is a free machine translation platform, developed by the TRANSDUCENS research group of University of Alicante. There is a basic web service, but the software can be locally installed and that is how <command>pomtrans</command> uses it (some distributions provide packages).</para>
 </listitem>
 </varlistentry>
 
 <varlistentry>
 <term>Google Translate (<literal>google</literal>)</term>
 <listitem>
 <para><ulink url="http://translate.google.com/">Google Translate</ulink> is Google's proprietary web machine-translation service. The user must obtain an API key from Google, and set it in <link linkend="sec-cmconfig">Pology configuration</link> under <literal>[pomtrans]/google-api-key</literal>. At the moment, <command>pomtrans</command> makes one query to the service per message, which can take quite some time on long PO files.</para>
 </listitem>
 </varlistentry>
 
 </variablelist>
 </para>
 
 </sect2>
 
 </sect1>
 
 </chapter>