<html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>Internationalization</title><meta name="generator" content="DocBook XSL Stylesheets V1.64.1"><link rel="home" href="index.html" title="OpenACS Core Documentation"><link rel="up" href="kernel-doc.html" title="Chapter�11.�Kernel Documentation"><link rel="previous" href="i18n-requirements.html" title="OpenACS Internationalization Requirements"><link rel="next" href="security-requirements.html" title="Security Requirements"><link rel="stylesheet" href="openacs.css" type="text/css"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><a href="http://openacs.org"><img src="/doc/images/alex.jpg" border="0" alt="Alex logo"></a><table width="100%" summary="Navigation header" border="0"><tr><td width="20%" align="left"><a accesskey="p" href="i18n-requirements.html">Prev</a> </td><th width="60%" align="center">Chapter�11.�Kernel Documentation</th><td width="20%" align="right"> <a accesskey="n" href="security-requirements.html">Next</a></td></tr></table><hr></div><div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="i18n"></a>Internationalization</h2></div></div><div></div></div><div class="authorblurb"><p>
      By <a href="http://www.petermarklund.com/" target="_top">Peter Marklund</a>
      and <a href="http://www.pinds.com/" target="_top">Lars Pind</a>
    </p>
          OpenACS docs are written by the named authors, and may be edited
          by OpenACS documentation staff.
        </div><div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="i18n-introduction"></a>Introduction</h3></div></div><div></div></div><p>
      This document describes how to develop internationalized OpenACS
      packages, including writing new packages with
      internationalization and converting old packages.  Text that
      users might see is "localizable text"; replacing monolingual text
      and single-locale date/time/money functions with generic
      functions is "internationalization"; translating first
      generation text into a specific language is "localization."  At
      a minimum, all packages should be internationalized.  If you do
      not also localize your package for different locales, volunteers
      may use a public "localization server" to submit suggested text.
      Otherwise, your package will not be usable for all locales.
    </p><p>
      The main difference between monolingual and internationalized
      packages is that all user-visible text in an internationalized
      package are coded as "message keys."  The message keys
      correspond to a message catalog, which contains versions of the
      text for each available language.  Both script files
      (ADP/TCL) and APM parameters are affected.
    </p><p>
      Other differences include: all dates read or written to the
      database must use internationalized functions.  All displayed
      dates must use internationalized functions.  All displayed
      numbers must use internationalized functions.
</p></div><div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="i18n-aolserver"></a>AOLserver Configuration for Multilingual Sites</h3></div></div><div></div></div><p>
      For multilingual websites we recommend using the UTF8
      charset. In order for AOLserver to use utf8 you need to set
      the config parameters OutputCharset and
      URLCharset to utf-8 in your AOLserver config file (use the etc/config.tcl
      template file). For sites running on Oracle you need to make
      sure that AOLserver is running with the NLS_LANG environment
      variable set to .UTF8. You should set this variable in the
      nsd-oracle run script (use the
      acs-core-docs/www/files/nds-oracle.txt template file).
    </p></div><div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="i18n-message-catalog"></a>Using the Message Catalog</h3></div></div><div></div></div><p>
      Localizable text must be handled in ADP files, in TCL files, and
      in APM Parameters.  OpenACS provides two approaches, message
      keys and localized ADP files.  For ADP pages which are mostly
      code, replacing the message text with message key placeholders
      is simpler.  This approach also allows new translation in the
      database, without affecting the file system.  For ADP pages
      which are static and mostly text, it may be easier to create a
      new ADP page for each language.  In this case, the pages are
      distinguished by a file naming convention.
    </p><div class="sect3" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="id2504281"></a>Separate Templates for each Locale</h4></div></div><div></div></div><p>If the request processor finds a file named <tt class="computeroutput">filename.locale.adp</tt>, where locale matches the user's locale, it will process that file instead of <tt class="computeroutput">filename.adp</tt>.  For example, for a user with locale <tt class="computeroutput">tl_PH</tt>, the file <tt class="computeroutput">index.tl_PH.adp</tt>, if found, will be used instead of <tt class="computeroutput">index.adp</tt>.  The locale-specific file should thus contain text in the language appropriate for that locale.  The code in the page, however, should still be in English.  Message keys are still processed.</p></div><div class="sect3" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="i18n-message-catalog-adps"></a>Message Keys in Template Files (ADP Files)</h4></div></div><div></div></div><p>
        Internationalizing templates is about replacing human readable
        text in a certain language with internal message keys, which
        can then be dynamically replaced with real human language in
        the desired locale.  Message keys themselves should be in
        ASCII English, as should all code.  Three different syntaxes
        are possible for message keys.
      </p><p>
        "Short" syntax is the recommended syntax and should be used
        for new development.  When internationalizing an existing
        package, you can use the "temporary" syntax, which the APM can
        use to auto-generate missing keys and automatically translate
        to the short syntax.  The "verbose" syntax is useful while
        developing, because it allows default text so that the page is
        usable before you have done
        localization.      </p><div class="itemizedlist"><ul type="disc"><li><p>
            The <span class="strong">short</span>:
            <tt class="computeroutput">#<span class="replaceable"><span class="replaceable">package_key.message_key</span></span>#</tt>
          </p><p>
            The advantage of the short syntax is that it's short. It's
            as simple as inserting the value of a variable.  Example:
            <span class="replaceable"><span class="replaceable">#forum.title#</span></span>
          </p></li><li><p>
            The <span class="strong">verbose</span>: <tt class="computeroutput">&lt;trn
            key="<span class="replaceable"><span class="replaceable">package_key.message_key</span></span>"
            locale="<span class="replaceable"><span class="replaceable">locale</span></span>"&gt;<span class="replaceable"><span class="replaceable">default
            text</span></span>&lt;/trn&gt;</tt>
          </p><p>
            The verbose syntax allows you to specify a default text in
            a certain language. This syntax is not recommended
            anymore, but it can be convenient for development, because
            it still works even if you haven't created the message
            in the message catalog yet, because what it'll do is
            create the message key with the default text from the tag
            as the localized message.  Example: <span class="emphasis"><em>&lt;trn
            key="forum.title" locale="en_US"&gt;Title&lt;/trn&gt;</em></span>
          </p></li><li><p>
            The <span class="strong">temporary</span>:
           <tt class="computeroutput"> &lt;#<span class="replaceable"><span class="replaceable">message_key</span></span>
           <span class="replaceable"><span class="replaceable">original text</span></span>#&gt;</tt>
          </p><p>
            This syntax has been designed to make it easy to
            internationalize existing pages. This is not a syntax that
            stays in the page. As you'll see later, it'll be replaced
            with the short syntax by a special feature of the APM. You
            may leave out the message_key by writing an underscore (_)
            character instead, in which case a message key will be
            auto-generated by the APM.  Example: <span class="emphasis"><em>&lt;_ Title&gt;</em></span>
          </p></li></ul></div><p>
        We recommend the short notation for new package development.
      </p></div><div class="sect3" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="i18n-message-catalog-tcl"></a>Message Keys in TCL Files</h4></div></div><div></div></div><p>
        In adp files message lookups are typically done with the syntax 
        <tt class="computeroutput">\#package_key.message_key\#</tt>. In Tcl
        files all message lookups *must* be on either of the following formats:
      </p><p>
        </p><div class="itemizedlist"><ul type="disc"><li>Typical static key lookup: <tt class="computeroutput">[_ package_key.message_key]</tt> - The message key and package key used here must be string literals, they can't result from variable evaluation.
          </li><li>
            Static key lookup with non-default locale: <tt class="computeroutput">[lang::message::lookup $locale package_key.message_key]</tt> - The message key and package key used here must be string literals, they can't result from variable evaluation.
          </li><li>
            Dynamic key lookup: 
<pre class="screen">
<b class="userinput"><tt>[lang::util::localize $var_with_embedded_message_keys]</tt></b>
- In this case the message keys in the variable <tt class="computeroutput">var_with_embedded_message_keys</tt> must appear as string literals <tt class="computeroutput">\#package_key.message_key\#</tt> somewhere in the code. Here is an example of a dynamic lookup:           
<b class="userinput"><tt>set message_key_array {
  dynamic_key_1  \#package_key.message_key1\#
  dynamic_key_2  \#package_key.message_key2\#
}

set my_text [lang::util::localize $message_key_array([get_dynamic_key])]
</tt></b>
</pre></li></ul></div><p>  
      </p><p>
        Translatable texts in page TCL scripts are often found in page titles,
        context bars, and form labels and options. Many times the texts are
        enclosed in double quotes. The following is an example of grep commands 
        that can be used on Linux to highlight translatable text in TCL files:
      </p><pre class="screen">
# Find text in double quotes
<b class="userinput"><tt>find -iname '*.tcl'|xargs egrep -i '"[a-z]'</tt></b>

# Find untranslated text in form labels, options and values
<b class="userinput"><tt>find -iname '*.tcl'|xargs egrep -i '\-(options|label|value)'|egrep -v '&lt;#'|egrep -v '\-(value|label|options)[[:space:]]+\$[a-zA-Z_]+[[:space:]]*\\?[[:space:]]*$'</tt></b>

# Find text in page titles and context bars
<b class="userinput"><tt>find -iname '*.tcl'|xargs egrep -i 'set (title|page_title|context_bar) '|egrep -v '&lt;#'</tt></b>

# Find text in error messages
<b class="userinput"><tt>find -iname '*.tcl'|xargs egrep -i '(ad_complain|ad_return_error)'|egrep -v '&lt;#'</tt></b>

      </pre><p>
        You may mark up translatable text in TCL library files and TCL pages
        with temporary tags on the &lt;#key text#&gt; syntax. 
        If you have a sentence or paragraph of text with
        variables and or procedure calls in it you should in most cases 
        try to turn the whole text into one
        message in the catalog (remember that translators is made easier the longer the phrases to translate are). In those cases, follow these steps:
      </p><div class="itemizedlist"><ul type="disc"><li>For each message call in the text, decide on a variable name and replace
            the procedure call with a variable lookup on the syntax %var_name%. Remember
            to initialize a tcl variable with the same name on some line above the text.</li><li>If the text is in a tcl file you must replace variable lookups 
            (occurences of $var_name or ${var_name}) with %var_name%</li><li>You are now ready to follow the normal procedure and mark up the text using a 
            tempoarary message tag (&lt;#_ text_with_percentage_vars#&gt;) and run the action 
            replace tags with keys in the APM.</li></ul></div><p>
      The variable values in the message are usually fetched with upvar, here is an example from dotlrn:
      </p><tt class="computeroutput">
      ad_return_complaint 1 "Error: A [parameter::get -parameter classes_pretty_name] 
                   must have &lt;em&gt;no&lt;/em&gt;[parameter::get -parameter class_instances_pretty_plural] to be deleted"
      </tt><p>
        was replaced by:
      </p><tt class="computeroutput">
      set subject [parameter::get -localize -parameter classes_pretty_name] 
      set class_instances [parameter::get -localize -parameter class_instances_pretty_plural]

      ad_return_complaint 1 [_ dotlrn.class_may_not_be_deleted]
      </tt><p>
      This kind of interpolation also works in adp files where adp variable values will be inserted into the message.
      </p><p>
      Alternatively, you may pass in an array list of the variable values to be interpolated into the message so that
      our example becomes:
      </p><pre class="screen">
<b class="userinput"><tt>set msg_subst_list [list subject [parameter::get -localize -parameter classes_pretty_name] class_instances [parameter::get -localize -parameter class_instances_pretty_plural]]

ad_return_complaint 1 [_ dotlrn.class_may_not_be_deleted $msg_subst_list]
</tt></b>
</pre><p>
        When we were done going through the tcl files we ran the following
        commands to check for mistakes:
      </p><pre class="screen">
# Message tags should usually not be in curly braces since then the message lookup may not be
# executed then (you can usually replace curly braces with the list command). Find message tags 
# in curly braces (should return nothing, or possibly a few lines for inspection)
<b class="userinput"><tt>find -iname '*.tcl'|xargs egrep -i '\{.*&lt;#'</tt></b>

# Check if you've forgotten space between default key and text in message tags (should return nothing)
<b class="userinput"><tt>find -iname '*.tcl'|xargs egrep -i '&lt;#_[^ ]'</tt></b>

# Review the list of tcl files with no message lookups
<b class="userinput"><tt>for tcl_file in $(find -iname '*.tcl'); do egrep -L '(&lt;#|\[_)' $tcl_file; done</tt></b>
</pre><p>
        When you feel ready you may vist your package in the
        <a href="/acs-admin/apm" target="_top">package manager</a>
        and run the action "Replace tags with keys
        and insert into catalog" on the TCL files that you've edited to
        replace the temporary tags with calls to the message lookup
        procedure.
      </p></div><div class="sect3" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="i18n-message-catalog-params"></a>Checking the Consistency of Catalog Files</h4></div></div><div></div></div><p>
        This section describes how to check that the set of keys used in
        message lookups in tcl, adp, and info files and the set of keys in
        the catalog file are identical.  The scripts below assume that
        message lookups in adp and info files are on the format
        \#package_key.message_key\#, and that message lookups in tcl files
        are always is done with one of the valid lookups described above. The script further assumes
        that you have perl installed and in your path.  Run the script like
        this:
      </p><tt class="computeroutput">
      acs-lang/bin/check-catalog.sh package_key
      </tt><p>
        where package_key is the key of the package that you want to
        test. If you don't provide the package_key argument then all
        packages with catalog files will be checked. 
        The script will run its checks primarily on en_US xml catalog files.
      </p></div><div class="sect3" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="i18n-message-apm-params"></a>APM Parameters</h4></div></div><div></div></div><p>
        Some parameters contain text that need to be localized. In
        this case, instead of storing the real text in the parameter,
        you should use message keys using the short notation above,
        i.e.  <span class="strong">#<span class="emphasis"><em>package_key.message_key</em></span>#</span>.
      </p><p>
       In order to avoid clashes with other uses of the hash
         character, you need to tell the APM that the parameter value
         needs to be localized when retrieving it. You do that by saying:
         <span class="strong">parameter::get -localize</span>.
      </p><p>
        Here are a couple of examples. Say we have the following two
        parameters, taken directly from the dotlrn package.
      </p><div class="informaltable"><table cellspacing="0" border="1"><colgroup><col><col></colgroup><thead><tr><th>Parameter Name</th><th>Parameter Value</th></tr></thead><tbody><tr><td>class_instance_pages_csv</td><td>#dotlrn.class_page_home_title#,Simple 2-Column;#dotlrn.class_page_calendar_title#,Simple 1-Column;#dotlrn.class_page_file_storage_title#,Simple 1-Column</td></tr><tr><td>departments_pretty_name</td><td>#departments_pretty_name#</td></tr></tbody></table></div><p>
        Then, depending on how we retrieve the value, here's what we get:
      </p><div class="informaltable"><table cellspacing="0" border="1"><colgroup><col><col></colgroup><thead><tr><th>Command used to retrieve Value</th><th>Retrieved Value</th></tr></thead><tbody><tr><td>parameter::get <span class="strong">-localize</span> -parameter class_instances_pages_csv</td><td>Kurs Startseite,Simple 2-Column;Kalender,Simple 1-Column;Dateien,Simple 1-Column</td></tr><tr><td>parameter::get <span class="strong">-localize</span> -parameter departments_pretty_name</td><td>Abteilung</td></tr><tr><td>parameter::get -parameter departments_pretty_name</td><td>#departments_pretty_name#</td></tr></tbody></table></div><p>
        The value in the rightmost column in the table above is the
        value returned by an invocation of parameter::get. Note that
        for localization to happen you must use the -localize flag.
      </p><p>
        The locale used for the message lookup will be the locale of
        the current request, i.e. lang::conn::locale or ad_conn
        locale.
      </p><p>
        Developers are responsible for creating the keys in the message
        catalog, which is available at <tt class="computeroutput">/acs-lang/admin/</tt>
      </p></div></div><div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="i18n-date-time-number"></a>Dates, Times, and Numbers</h3></div></div><div></div></div><p>
      Dates and times must be converted when stored in the database,
      when retrieved from the database, and when displayed.  All dates
      are stored in the database in the server's timezone, which is an
      APM Parameter set at
      <tt class="computeroutput">/acs-lang/admin/set-system-timezone</tt>
      and readable at
      <tt class="computeroutput">lang::system::timezone.</tt>.  When
      retrieved from the database and displayed, dates and times must
      be localized to the user's locale.
    </p><div class="orderedlist"><ol type="1"><li><p>
          Get the date in ANSI format from the database (YYYY-MM-DD
          HH24:MI:SS; the time portion is optional).  By convention,
          we identify dates in ansi format by ending the column name
          with <tt class="computeroutput">_ansi</tt>.
          Example:</p><pre class="programlisting">select to_char(posting_date, 'YYYY-MM-DD HH24:MI:SS') as posting_date_ansi
  from table
</pre></li><li><p>
          Use the Tcl command <tt class="computeroutput">lc_time_fmt</tt> to format the
          date in "pretty" format.  Several standard formats localize automatically:
</p><div class="itemizedlist"><ul type="disc"><li><p>
          %c: Long date and time (Mon November 18, 2002 12:00 AM)
        </p></li><li><p>
          %x: Short date (11/18/02)
        </p></li><li><p>
          %X: Time (12:00 AM)
        </p></li><li><p>
          %q: Long date without weekday (November 18, 2002)
        </p></li><li><p>
           %Q: Long date with weekday (Monday November 18, 2002)
        </p></li></ul></div><p>
      The "q" format strings are OpenACS additions; the rest follow unix standards (see <tt class="computeroutput">man
      strftime</tt>).
    </p><pre class="programlisting">set posting_date_pretty [lc_time_fmt $posting_date_ansi "%q"]</pre></li><li><p>
          Use the <tt class="computeroutput">*_pretty</tt> version in your ADP page.
        </p></li></ol></div><p>
      To internationalize numbers, use <tt class="computeroutput">lc_numeric $value</tt>, which formats the number using the appropriate decimal point and thousand separator for the locale.
    </p></div><div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="i18n-forms"></a>Internationalizing Forms</h3></div></div><div></div></div><p>When coding forms, remember to use message keys for each piece of text that is user-visible, including form option labels and button labels.</p></div><div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="i18n-convert"></a>Internationalizing Existing Packages</h3></div></div><div></div></div><div class="sect3" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="id2578783"></a>Internationalize Message text in ADP and TCL</h4></div></div><div></div></div><p>Acs-lang includes tools to automate some
        internationalization.  From
        <tt class="computeroutput">/acs-admin/apm/</tt>, select a
        package and then click on
        <tt class="computeroutput">Internationalization</tt>, then
        <tt class="computeroutput">Convert ADP, Tcl, and SQL files to using the
        message catalog.</tt>.</p></div><div class="sect3" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="id2578807"></a>Internationalize Package Parameters with visible messages</h4></div></div><div></div></div><p>
      See <a href="i18n.html#i18n-message-catalog-params" title="Checking the Consistency of Catalog Files">the section called &#8220;Checking the Consistency of Catalog Files&#8221;</a>
    </p></div><div class="sect3" lang="en"><div class="titlepage"><div><div><h4 class="title"><a name="id2584994"></a>Internationalize Date and Time queries</h4></div></div><div></div></div><div class="orderedlist"><ol type="1"><li><p>Find datetime in .xql files.  Use command line tools to find suspect SQL code:</p><pre class="programlisting">grep -r "to_char.*H" *
grep -r "to_date.*H" *
</pre></li><li><p>In SQL statements, replace the format string with the ANSI standard format, <tt class="computeroutput">YYYY-MM-DD HH24:MI:SS</tt> and change the field name to *_ansi so that it cannot be confused with previous, improperly formatting fields.  For example,</p><pre class="programlisting">to_char(timestamp,'MM/DD/YYYY HH:MI:SS') as foo_date_pretty</pre><p>becomes</p><pre class="programlisting">to_char(timestamp,'YYYY-MM-DD HH24:MI:SS') as foo_date_ansi</pre></li><li><p>In TCL files where the date fields are used, convert the datetime from local server timezone, which is how it's stored in the database, to the user's timezone for display.  Do this with the localizing function <tt class="computeroutput"><a href="/api-doc/proc-view?proc=lc_time_system_to_conn" target="_top">lc_time_system_to_conn</a></tt>:</p><pre class="programlisting">
set foo_date_ansi [lc_time_system_to_conn $foo_date_ansi]</pre><p>When a datetime will be written to the database, first convert it from the user's local time to the server's timezone with <tt class="computeroutput"><a href="/api-doc/proc-view?proc=lc%5ftime%5fconn%5fto%5fsystem" target="_top">lc_time_conn_to_system</a></tt>.
</p></li><li><p>When a datetime field will be displayed, format it using the localizing function <tt class="computeroutput"><a href="/api-doc/proc-view?proc=lc_time_fmt" target="_top">lc_time_fmt</a></tt>. lc_time_fmt takes two parameters, datetime and format code.  Several format codes are usable for localization; they are placeholders that format dates with the appropriate codes for the user's locale.  These codes are: <tt class="computeroutput">%x, %X, %q, %Q, and %c.</tt></p><pre class="programlisting">set foo_date_pretty [lc_time_fmt $foo_date_ansi "%x %X"]</pre></li></ol></div></div></div><div class="sect2" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="i18n-design"></a>Design Notes</h3></div></div><div></div></div><p>User locale is a property of ad_conn, <tt class="computeroutput">ad_conn locale</tt>.  The request processor sets this by calling <tt class="computeroutput">lang::conn::locale</tt>, which looks for the following in order of precedence:</p><div class="orderedlist"><ol type="1"><li><p>Use user preference for this package (stored in ad_locale_user_prefs)</p></li><li><p>Use system preference for the package (stored in apm_packages)</p></li><li><p>Use user's general preference (stored in user_preferences)</p></li><li><p>Use Browser header (<tt class="computeroutput">Accept-Language</tt> HTTP header)</p></li><li><p>Use system locale (an APM parameter for acs_lang)</p></li><li><p>default to en_US</p></li></ol></div><p>For ADP pages, message key lookup occurs in the templating engine.  For TCL pages, message key lookup happens with the <tt class="computeroutput">_</tt> function.  In both cases, if the requested locale is not found but a locale which is the default for the language which matches your locale's language is
found, then that locale is offered instead.</p></div></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="i18n-requirements.html">Prev</a> </td><td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td><td width="40%" align="right"> <a accesskey="n" href="security-requirements.html">Next</a></td></tr><tr><td width="40%" align="left">OpenACS Internationalization Requirements </td><td width="20%" align="center"><a accesskey="u" href="kernel-doc.html">Up</a></td><td width="40%" align="right"> Security Requirements</td></tr></table><hr><address><a href="mailto:docs@openacs.org">docs@openacs.org</a></address></div><a name="comments"></a><center><a href="http://openacs.org/doc/i18n.html#comments">View comments on this page at openacs.org</a></center></body></html>