ACS 4 Globalization Detailed Design

by Henry Minsky

I. Essentials

When applicable, each of the following items should receive its own link:

User directory: none
ACS administrator: acs-lang
Subsite administrator directory: none
Tcl script directory: /api-doc
PL/SQL file:
Data model:
Requirements document

II. Introduction

III. Historical Considerations

V. Design Tradeoffs

Performance: availability and efficiency
For Internationalization to be effective, it needs to be integrated into every module in the system. Thus making the overhead as low as possible is a priority, otherwise developers will be reluctant to use it in their code.
Wherever possible, caching in AOLserver shared memory is used to remove the need to touch the database. Precompiling of template files should reduce the overhead to zero in most cases for translation message lookups. The amount of overhead added to the request processor can be reduced by caching filesystem information on matching of template files for locales.
Flexibility
Interoperability
Reliability and robustness
Usability

Areas of interest to developers:

Maintainability
Portability
The ACS Tcl I18N APIs should be as close as possible to the ultimate Java APIs. This means that using the same templates if possible, as well as the same message catalogs and format strings should be a strong goal.
Reusability
Testability
A set of unit tests are included in the acs-lang package, to allow automatic testing after installation.

VI. API

VI.A Locale API

10.30 A Locale object represents a specific geographical, political, or cultural region. An operation that requires a Locale to perform its task is called locale-sensitive and uses the Locale to tailor information for the user. For example, displaying a number is a locale-sensitive operation--the number should be formatted according to the customs/conventions of the user's native country, region, or culture.

We will refer to a Locale by a combination of a language and country. In the Java Locale API there is an optional variant which can be added to a locale, which we will omit in the Tcl API.

The language is a valid ISO Language Code. These codes are the lower-case two-letter codes as defined by ISO-639. You can find a full list of these codes at a number of sites, such as:
http://www.ics.uci.edu/pub/ietf/http/related/iso639.txt

The country is a valid ISO Country Code. These codes are the upper-case two-letter codes as defined by ISO-3166. You can find a full list of these codes at a number of sites, such as:
http://www.chemie.fu-berlin.de/diverse/doc/ISO_3166.html

Examples are

en_US English US
ja_JP Japanese
fr_FR France French.

The i18n module figures out the locale for a current request makes it accessible via the ad_locale function:

[ad_locale user locale] => fr_FR
[ad_locale subsite locale] => en_US

It has not yet been decided how the user's preferred locale will be initialized. For now, there is a site wide default package parameter [parameter::get -parameter DefaultLocale -default "en_US"], and an API for setting the locale with the preference stored in a session variable: The ad_locale_set function is used to set the user's preferred locale to a desired value. It saves the value in the current session.

    ad_locale_set locale "en_US"
       will also automatically set [ad_locale user language]
          ( to "en" in this case)

    ad_locale_set timezone "PST"

The request processor should use the ad_locale API to figure out the preferred locale for a request (perhaps combining user preference with subsite defaults in some way). It will make this information accesible via the ad_conn function:

    ad_conn locale

Character Sets and Encodings

We refer to MIME character set names which are the valid values which can be passed in a MIME header, such as

Content-Type: text/html; charset=iso-8859-1

You can obtain the preferred character set for a locale via the ad_locale API shown below:

set locale "en_US"
[ad_locale charset $locale] => "iso-8859-1" or "shift_jis"

Returns a case-insensitive name of a MIME character set.

We already have an AOLserver function to convert a MIME charset name to a Tcl encoding name:

[ns_encodingforcharset "iso-8859-1"] => iso8859-1

Templating

The goal of templates is to separate program logic from data presentation.

For presenting data in multiple languages, there are two basic ways to use templates for a given abstract URL. Say we have the URL "foo", for example. We can provide templates for it in the following ways:

Separate template for each target language
Have a copy of each template file in each language you support, e.g., foo.en.adp, foo.fr.adp, foo.de.adp, etc.
We will refer to this style of template pages as language-specific templates.
A single template file for multiple languages
You write your template to contain references to translation strings either from data sources or using <TRN> tags.
For example, a site might support multiple languages, but use a single file foo.adp which contains no language-specific content, and would only make use of data sources or <TRN> tags which in turn use the message catalog to look up language-specific content.
We will refer to this style of page as a multilingual template.

Both styles of authoring templates will probably be used; For pages which contain a lot of free form text content, then having a separate template page for each language would be easiest.

But for a page which has a very fixed format, such as a data entry form, it would mean a lot less redundant work to use a single template source page to handle all the languages, and to have all language-dependent strings be looked in a message catalog. We can do this either by creating data sources which call lang_message_lookup, or else use the <TRN> tag to do the same thing from within an ADP file.

Caching multilingual ADP Templates

Message catalog lookups can be potentially expensive, if many of them are done in a page. The templating system can already precompile and and cache adp pages. This works fine for a page in a specific language such as foo.en.adp, but we need to modify the caching mechanism if we want to use a single template file to target multiple languages.

Computing the Effective Locale

Let's say you have a template file "foo.adp" and it contains calls to look up message strings using the TRN tag:

<master>
<trn key=username_prompt>Please enter your username</tr>
<input type=text name=username>
<p>
<trn key=password_prompt>Enter Password:</trn>
<input type=password name=passwd>

If the user requests the page foo, and their ad_locale is "en_US" then effective locale is "en_US". Message lookups are done using the effective locale. If the user's locale is "fr_FR", then the effective locale will be "fr_FR".

If we evaluate the TRN tags at compile time then we need to associate the effective locale in which the page was evaluated with the cached compiled page code.

The effective locale of a template page that has an explicit locale, such as a file named "foo.en.adp" or "foo.en_US.adp", will be that explicit locale. So for example, even if a user has a preferred locale of "fr_FR", if there is only a page named "foo.en.adp", then that page will be evaluated (and cached) with an effective locale of en_US.

VI.B Naming of Template Files To Encode Language and Character Set

10.40 The templating system will use the Locale API to obtain the preferred locale for a page request, and will attempt to find a template file which most closely matches that locale.

We will use the following convention for naming template files: filename.locale_or_language.adp.

Examples:

foo.en_US.adp
foo.en.adp

foo.fr_FR.adp
foo.fr.adp

foo.ja_JP.adp
foo.ja.adp

The user request has a locale which is of the form language_country. If someone wants English, they will implicitly be choosing a default, such as en_US or en_GB. The default locale for a language can be configured in the system locale tables. So for example the default locale for "en" could be "en_US".

The algorithm for finding the best matching template for a request in a given locale is given below:

Find the desired target locale using [ad_conn locale] NOTE: This will always be a specific Locale (i.e., language_COUNTRY)
Look for a template file whose locale suffix matches exactly.
For example, if the filename in the URL request is simply foo and [ad_conn locale] returns en_US then look for a file named foo.en_US.adp.
If an exact match is not found, look for template files whose name matches the language portion of the target locale.
For example, if the URL request name is foo and [ad_conn locale] returns en_US and a file named foo.en_US.adp is not found, then look for all templates matching "en_*" as well as any template which just has the "en" suffix.
So for example if the user's locale en_GB and the following files exist:
foo.en_US.adp
then use foo.en_US.adp
If however both foo.en_US.adp and foo.en.adp exist, then use foo.en.adp preferentially, i.e., don't switch locales if you can avoid it. The reasoning here is that people can be very touchy about switching locales, so if there is a generic matching language template available for a language, use it rather than using an incorrect locale-specific template.
If no locale-specific template is found, look for a template matching just the language
I.e., if the request is for en_US, and there exists a file foo.en.adp, use that.
If no locale-specific template is found, look for a simple .adp file, such as foo.adp.

Once a template file is found we must decide what character set it is authored in, so that we can correctly load it into Tcl (which converts it to UTF8 internally).

It would be simplest to mandate that all templates are authored in UTF8, but that is just not a practical thing to enforce at this point, I believe. Many designers and other people who actually author the HTML template files will still find it easier to use legacy tools that author in their "native" character sets, such as ShiftJIS in Japan, or BIG5 in China.

So we make the convention that the template file is authored in it's effective locale's character set. For multilingual templates, we will load the template in the site default character set as specified by the AOLserver OutputCharset initializatoin parameter. For now, we will say that authoring generic multilingual adp files can and should be done in ASCII. Eventually we can switch to using UTF8.

A character set corresponding to a locale can be found using the [ad_locale charset $locale] command. The templating system should call this right after it computes the effective locale, so it can set up that charset encoding conversion before reading the template file from disk.

We read the template file using this encoding, and set the default output character set to it as well. Inside of either the .adp page or the parent .tcl page, it is possible for the developer to issue a command to override this default output character set. The way this is done is currently to stick an explicit content-type header in the AOLserver output headers, for example to force the output to ISO-8859-1, you would do

ns_set put [ns_conn outputheaders] "content-type" "text/html; charset=iso-8859-1"

design questionWe should have an API for this. The hack now is that the adp handler adp_parse_ad_locale user_file looks at the output headers, and if it sees a content type with an explicit charset, it passes it along to ns_return.

The default character set for a template .adp file should be the default system encoding.

VI.C Loading Regular Tcl Script Files

10.50 By default, tcl and template files in the system will be loaded using the default system encoding. This is generally ISO-8859-1 for AOLserver running on Unix systems in English.

This default can be overridden by setting the AOLserver init parameter for the MIME type of .tcl files to include an explcit character set. If an explicit MIME type is not found, ns_encodingfortype will default to the AOLserver init parameter value DefaultCharset if it is set.

Example AOLserver .ini configuration file to set default script file and template file charset to ShiftJIS:

ns_section {ns/mimetypes }
...
ns_param .tcl {text/plain; charset=shift_jis}
ns_param .adp {text/html; charset=shift_jis}

ns_section ns/parameters
...
# charset hacking
ns_param HackContentType 1
ns_param URLCharset shift_jis
ns_param OutputCharset shift_jis
ns_param HttpOpenCharset shift_jis
ns_param DefaultCharset shift_jis

VI.A Message Catalog API

We want to use something like the Java ResourceBundle, where the developer can declare a set of resources for a given namespace and locale.

For AOLserver/TCL, to make the message catalog more manageable, we will split it into one message catalog per package, plus one default global message namespace in case we need it. So for example,

Message lookups are done using a combination of a key string and a locale or language, as well as an implicit package prefix on the key string. The API for using the message catalog is as follows:

lang_message_lookup locale key [default_string]
lang_message_lookup is abbreviated by the procedure named "_", which is the convention used by the GNU strings message catalog package.

The locale arg can actually be a full locale, or else a simple language abbrev, such as fr, en, etc. The lookup rules for finding strings based on key and locale are tried in order as follows:

Lookup is first tried with the full locale (if present) and package.key
Lookup is tried with just the language portion of the locale and package.key
Lookup is tried with the full locale and key without package prefix.
Lookup is tried with language and key without package prefix.

Example: You are looking up the message string "Title" in the notes package.

[lang_message_lookup $locale notes.title "Title"]

can be abbreviated by
[_ $locale notes.title "Title"]

# message key "title" is implicitly with respect to package key
#  "notes", i.e., notes.title
[_ $locale title "Title"]

The string is looked up by the symbolic key notes.title (or title for short), and the constant value "Title" is supplied as documentation and as a default value. Having a default value allows developers to code their application immediately without waiting to populate the message catalog.

Default Package Namespace

By default, keys are prefixed with the name of the current package (if a page request is being processed). So a lookup of the key "title" in a page in the bboard package will actually reference the "bboard.title" entry in the message catalog.

You can override this behavior by either using a fully qualified key such as bboard.title or else by changing the message catalog namespace using the lang_set_package command:

[lang_set_package "bboard"]

So for example code that runs in a scheduled proc, where there is not necessarily any concept of a "current package", would either use fully qualified keys to look up messages, or else call lang_set_package before doing a message lookup.

Message Catalog Definition Files

A message catalog is defined by placing a file in the catalog subdirectory of a package. Each file defines a set of messages in different locales, and the file is written in a character set specified by it's file suffix:

/packages/bboard/catalog/
			 bboard.iso-8859-1
			 bboard.shift_jis
			 bboard.iso-8859-6

A message catalog file consists of tcl code to define messages in a given language or locale:


_mr en mail_notification "This is an email notification"
_mr fr mail_notification "Le notification du email"
...

In the example above, if the catalog file was loaded from the bboard package, all of the keys would be prefixed autmatically with "bboard.".

Loading A Message Catalog At Package Init Time

The API function

lang_catalog_load package_key

Is used to load the message catalogs for a package. The catalog files are stored in a package subdirectory called catalog. Their file names have the form *.encoding.cat, where encoding is the name of a MIME charset encoding (not a Tcl charset name as was used in a previous version of this command).

/packages/bboard/catalog
                        /main.iso8859-1.cat
                        /main.shift_jis.cat
                        /main.iso-8859-6.cat
                        /other.iso8859-1.cat
                        /other.shift_jis.cat
                        /other.iso-8859-6.cat

You can add more pseudo-levels of hierarchy in naming the message keys, using any separator character you want, for example

_mr fr alerts.mail_notification "Le notification du email"

which will be stored with the full key of bboard.alerts.mail_notification.

Calling the Message Catalog API from inside of Templates

Inside of a template, you can always make a call to the message catalog API via a Tcl escape:

<%= [_ $locale bboard.passwordPrompt "Enter Password"]%>

However, this is awkward and ugly to use. We have defined an ADP tag which invokes the message catalog lookup. As explained in the previous section, since our system precompiles adp templates, we can get a performance improvement if we can cache the message lookups at template compile time.

The <TRN> tag is a call to lang_message_lookup that can be used inside of an ADP file. Here is the documention:

Procedure that gets called when the <trn> tag is encountered on an ADP page. The purpose of the procedure is to register the text string enclosed within a pair of <trn> tags as a message in the catalog, and to display the appropriate translated string. Takes three optional parameters: lang, type and key.

key specifies the key in the message catalog. If it is omitted this procedure returns simply the text enclosed by the tags.
lang specifies the language of the text string enclosed within the flags. If it is ommitted value defaults to English.
type specifies the context in which the translation is made. If omitted, type is user which means that the translation is provided in the user's preferred language.
static specifies that this tag should be translated once at templat compile time, rather than dynamically every time the page is run. This will be unneccessaru and will be deprecated once we have implemented effective locale based cacheing for templates.
Example 1: Display the text string Hello on an ADP page (i.e. do nothing special):
    <trn>Hello</trn>
    
Example 2: Assign the key key hello to the text string Hello and display the translated string in the user's preferred language:
    <trn key="hello">Hello</trn>
    
Example 3: Specify that Bonjour needs to be registered as the French translation for the key hello (in addition to displaying the translation in the user's preferred language):
    <trn key="hello" lang="fr">Bonjour</trn>
    
Example 4: Register the string and display it in the preferred language of the current user. Note that the possible values for the type paramater are determined by what has been implemented in the ad_locale procedure. By default, only the user type is implemented. An example of a type that could be implemented is subsite, for displaying strings in the language of the subsite that owns the current web page.
    <trn key="hello" type="user">Hello</trn>
    
Example 5: Translates the string once at template compile time, using the effective local of the page.
    <trn key="hello" static>Hello</trn>
    

VII. Data Model Discussion

Internationalizing the Data Models

Some data which is stored in ACS package and core database tables may be presented to users, and thus may need to be stored in multiple languages. Examples of this are the descriptions of package or site parameters in the administrative interface, the "pretty names" of objects, and group names.

Tables which are in acs kernel and have user-visible names that may need to be translated in order to create an admin back end in another language:

user groups:
   group_name

acs_object_types:
   pretty_name
   pretty_plural

acs_attributes:
   pretty_name
   pretty_plural

acs_attribute_descriptions
   description (clob)

procedure add_description- add a lang arg ?

acs_enum_values ? pretty_name

acs_privileges: 
  pretty_name
  pretty_plural

apm_package_types
  pretty_name
  pretty_plural


apm_package "instance_name"? Maybe a given instance
gets instantiated with a name in the desired language?


apm_parameters: 
   parameter_name
   section_name

One approach is to split a table into two tables, one holding language-independent datam, and the other holding language-dependent data. This approach was described in the ASJ Multilingual Site Article.

In that case, it is convenient to create a new view which looks like the original table, with the addition of a language column that you can specify in the queries.

Drawbacks to Splitting Tables

It is not totally transparent to developers
Every query against the table which requests or modifies language-dependent columns must now include a WHERE clause to select the language.

Extra join may slow things down
The extra join of the two tables may cause queries to slow down, although I am not sure what the actual performance hit might be. It shouldn't be too large, because the join is against a fully indexed table.

VIII. User Interface

IX. Configuration/Parameters

X. Code Examples

fconfigure -encoding blah

content type in outputheaders set for encoding conversion

ad_proc adp_parse_ad_conn_file {} {
    handle a request for an adp and/or tcl file in the template system.
} {
    namespace eval template variable parse_level ""
    #ns_log debug "adp_parse_ad_conn_file => file '[file root [ad_conn file]]'"
    set parsed_template [template::adp_parse [file root [ad_conn file]] {}]
    db_release_unused_handles
    if {![empty_string_p $parsed_template]} {
        set content_type [ns_set iget [ns_conn outputheaders] "content-type"]
        if { $content_type == "" } {
	    set content_type  [ns_guesstype [ad_conn file]]
        } else {
            ns_set idelkey [ns_conn outputheaders] "content-type"
        }
	ns_return 200 $content_type $parsed_template
    }
}

XI. Future Improvements/Areas of Likely Change

XII. Authors

System creator
System owner: hqm@arsdigita.com
Documentation author: hqm@arsdigita.com

XII. Revision History

The revision history table below is for this template - modify it as needed for your actual design document.

Document Revision #	Action Taken, Notes	When?	By Whom?
0.1	Creation	12/4/2000	Henry Minsky
0.2	More specific template search algorithm, extended message catalog API to use package keys or other namespace	12/4/2000	Henry Minsky
0.3	Details on how the <TRN> tag works in templates	12/4/2000	Henry Minsky
0.4	Definition of effective locale for template caching, documentation of TRN tag	12/12/2000	Henry Minsky

hqm@arsdigita.com