Translations in Ubuntu (from a developer perspective)

This document tries to outline:

• How Ubuntu deals with translations (and i18n in general) from a developer's perspective
• Which tools are interesting for developers to make sure that translations and i18n work fine

General information

Glossary of i18n-related terms

gettext

Translation domain (domain)
The translation domain is a unique string assigned by the programmer in the code (usually in the build system) and used by the gettext functions to locate the message catalog where translations will be loaded from. The general form to compute the catalog's location is:

dir_name/locale/LC_category/domain_name.mo

which in Ubuntu expands to either /usr/share/locale/$LOCALE/LC_MESSAGES/domain_name.mo for translations not included in language packs (not in main) or /usr/share/locale-langpack/$LOCALE/LC_MESSAGES/domain_name.mo for those shipped in language packs. $LANGUAGE is generally the ISO 639-1 2-letter code (e.g. 'ca' or 'de') or ISO 639-2 3-letter code (e.g. 'ast') for the particular language. As an example, when using Nautilus in a Catalan locale with Ubuntu, the gettext functions will look for the message catalogue at:

/usr/share/locale-langpack/ca/LC_MESSAGES/nautilus.mo

The corresponding translation template should have the same translation domain in its filename, e.g. nautilus.pot. The translation domain must be unique across all applications and packages. I.e. something generic like messages.pot won't work.

Translation template (POT file, PO template file)
This term is used in Launchpad Translations to identify a set of messages that need to be translated (and sometimes also the set of translations for them). These messages are (for gettext packages) defined in a POT file in a source package. A source package using gettext may have one or more POT files. Each POT file has a unique name that specifies its 'translation domain' (see above). The POT file name becomes the template name in LP (without the ".pot" extension). Non-gettext packages do not use POT files, for example Mozilla uses XPI files, but these may still have LP templates that are imported using other means. A package's POT files are generated or updated at package build time or manually, after which they contain the package's latest set of translatable messages. Then, the POT files are imported into LP Translations at package upload time and, optionally, later through other LP functions. On import, if a template already exists in LP, the LP data for it is updated to reflect the new set of translatable messages from the corresponding POT file, otherwise LP adds a new template. Each template usually also has translations associated with it in LP. Thus, the term "template" is often used to indicate all of this taken together, namely, the set of translatable messages for a specific translation domain and all the corresponding per-language translations.

Note that "template" in this LP sense corresponds to a POT file in a source package sense, but the two are not identical. Bear in mind that Launchpad Translations has a database into which messages and translations are imported and from which they are exported. POT files and PO files are not stored in the database, but are generated as needed.

Translation (PO file)
Source packages using gettext contain translations (for a given translation domain and for each language) in dedicated PO files. PO file names are based on the language code, for example: fr.po for French, or pt_BR.po for Portuguese/Brazil. These PO files are imported to Launchpad Translations on package upload and at other times using other LP functions. Their translations are merged with any existing translations for the target template (that is, for the translation domain) in LP. The results of this merge depend on the context (in some cases, the new translations appear as suggestions, in other cases, they are considered the upstream translations and take precedence over existing LP translations (unless specific messages were intentionally modified in LP). Once imported, translations are displayed in the Launchpad Translations web UI. As with POT files, PO files can be exported from LP Translations. (Note that PO files are not saved in the LP database, rather they are generated from the database on export as needed.) For example, PO files can be downloaded in a tarball, and they can be automatically exported once a day to a specified bzr branch. Such exporting allows project maintainers to easily integrate translations done in Launchpad back into a source package for later use at run time. (Note that this reintegration is not required for packages in Ubuntu Main because these packages derive their translations from language packs, which are language-specific exports of LP Translations.)

Message catalog (MO file)

intltool

Imports queue

Packaging

Language Packs

General

Packages in main and restricted don't contain translations (*.mo) files themselves, they are stripped during the build on the Launchpad buildds and put into language-pack-* packages instead. This is done in an attempt to bundle the translations you're interested in and cut down the occupied space.

Translation templates

For pkgstriptranslations (in the pkgbinarymangler package) to do the job right and translations to be imported into Launchpad, please make sure that your package in main or restricted generates a POT translation template during the build. It does not necessarily need to be shipped in the source or in the binary package. Generating it during the build is good enough.

This can be achieved by running the following in debian/rules:

• For packages using CDBS and automake:

  • You just need to include /usr/share/cdbs/1/rules/langpack.mk in debian/rules

  • If the package is a GNOME one, and you are using the gnome.mk CDBS class already, you should be good to go, since it already includes langpack.mk

  • KDE packages: TODO
  • If for some reason the POT template cannot be generated during the build, see the sections below.

• Other packages, using intltool usually require:

  • A call to intltool-update -p in debian/rules, or

  • a call to an already existing Makefile target to build the POT (this does exist quite often in fact).

• Packages which don't use intltool:

  • Need to get fixed individually

  • Usually by adding a call to xgettext and po/POTFILES.in, or

  • Finding an already existing Makefile target. e.g. Use $(MAKE) -C po update-po instead of (cd po; make update-po; cd ..).

• Packages which use python-distutils-extra:

  • Template creation should happen automagically

  • TODO: explanation of the command and auto modules

Other translation files

Desktop Entries

Translations of desktop entries (.desktop files) are also stripped out and included in language packs.

Contrary to what other distros do (except currently OpenSUSE), the translations are not expected to be in the .desktop files themselves (static translations), but rather in the same .mo file of the application, from which they are then loaded at runtime. We do this in order to be able to translate them in Launchpad and to ship them in language packs, so that they can be edited and updated by Ubuntu translators.

The runtime loading is done by a patch in glib (or in kdelibs for Kubuntu), which recognises the the X-GNOME-Gettext-Domain additional key we add to all Ubuntu .desktop files and adds gettext support when loading them. Note that if the .desktop file already contains translations, these static translations will be preferred.

As an example for this key in action, here's how it is done in the Transmission source package in Jaunty (from the 20_add_X-Ubuntu-Gettext-Domain.diff patch):

Index: transmission-1.40-2ubuntu1/gtk/transmission.desktop.in
===================================================================
--- transmission-1.40-2ubuntu1.orig/gtk/transmission.desktop.in 2008-12-09 18:57:13.000000000 +0000
+++ transmission-1.40-2ubuntu1/gtk/transmission.desktop.in      2008-12-09 18:58:54.000000000 +0000
@@ -9,3 +9,4 @@
 Type=Application
 MimeType=application/x-bittorrent;
 Categories=Network;FileTransfer;P2P;GTK;
+X-GNOME-Gettext-Domain=transmission

Also note that X-GNOME-Gettext-Domain used to be called X-Ubuntu-Gettext-Domain in the past. The Ubuntu glib version supports both (it first checks for X-GNOME- and then X-Ubuntu-)

In summary, for packages in main we strip the translations out of the .desktop files, and glib/kdelibs carry a distro patch to load translations from the .mo file. They basically use the X-Ubuntu-Gettext-Domain key to figure out where to get the translation from.

Additional information:

• Original specification

• Original specification for KDE

Schema files

Policy files

Interactive upgrade hooks

After certain upgrades (most notably upgrades between Ubuntu releases), there are follow-up actions which should be taken by the user. These are not logically part of the upgrade, but should be dealt with at some point after the upgrade is complete.

They take the form of helper scripts that can be called by the package's postinst script, and can be internationalized through language packs.

Here's an example showing how these can be internationalized:

_Name: The great UTF-8 Migration
Priority: Medium
Command: bc
Terminal: True
GettextDomain: hwdb-clinet
DisplayIf: shell-command
Title: Change the default title here
DontShowAfterReboot: False
_Description: 
 This command will convert your stuff into UTF-8.
 Use this command if you want a working gnome desktop and you
 feel the world should be a better place.

[type: gettext/rfc822deb] debian/utf8-migration.installnote

install -d /var/lib/update-notifier/user.d/
sed 's/^_//' /usr/share/utf8-migration-tool/utf8-migration.installnote > /var/lib/update-notifier/user.d/utf8-migration

Verifying translation uploads

This information is intended to help package maintainers to optimize their packages, so that translation related files get automatically imported into Launchpad.

Testing the package locally

To test if your package conforms to the requirements of the import script in Launchpad, please do the following:

1. Install the pkgbinarymangler package

2. In /etc/pkgbinarymangler/striptranslations.conf set enable: to true

3. Optionally, install any build dependencies: sudo apt-get build-dep $PKGNAME

4. Build the package with debuild -uc -us -b

5. You will find a <pkgname>-translations.tar.gz tarball along with the .deb package(s)

6. Untar that tarball into a temporary directory
7. Compare if the directory structure and the .pot and .po files conform to the requirements listed below

Guidelines

If the application uses gettext, like most GTK and GNU applications do:

• Template name and domain: the filename of the template (.pot file) should be the same as what the expected translation domain is.

• Translation filenames: the actual translations (.po files) must have a filename of the format $LANGUAGECODE.po or $LOCALECODE.po (e.g. de.po, pt_BR.po, zh_TW.po)

• Template and translations location: all translation-related files (the .pot template and its corresponding .po templates) must end up in the same (sub-)directory if possible.

• Multiple domains: if the application has multiple translation domains (i.e. multiple .pot files), each .pot file and their corresponding .po files should end up in a separate subdirectory (e.g. po/nautilus/nautilus.pot and po/libnautilus/libnautilus.pot)

• Recommended translations layout: it is recommended that templates and translations for the main application or library are located in the po/ directory.

• Recommended help layout: it is recommended that templates and translations for help files are located in the help/po/ directory. Note: they will be ignored by Launchpad, since we currently have no infrastructure to deal with them properly.

• Recommended documentation layout: it is recommended that templates and translations for documentation files are located in the doc/po/ directory. Note: they will also be ignored by Launchpad for the same reason as the help translations.

• Obsolete translations: if there are .po files present in the package but no associated .pot template, please remove them, since they cannot be used without a .pot template. There might be a bug in the package or in the source code whereby the .pot file doesn't get generated, so make sure to double-check that as well.

• Valid POT template: please check if the .pot file has any meaningful content. Empty .pot files should be removed and the package build rules be fixed.

• Templates generated by patches: if your package contains patches and those patches result into extra templates, like patches.pot, please merge those changes into the main template, as we have no use for patch .pot files. Such .pot files should not exist in the final tarball.

• Multiple templates merged into one: if the package generates multiple .pot files and later merges them into one, only the final .pot file should be present in the final tarball.

• Architecture-dependent templates: if the package creates the same .pot file in architecture-dependent subdirectories, please consider to merge them, as we can only deal with one single .pot per translation domain.

• Test templates: test routines and test files should not generate any .pot template or ship any .po file. If such files are present, please remove them.

• Use UTF-8 encoding: if you have multiple .po files for a language, each in a different encoding, please consider renaming the ones encoded in UTF-8 to be the main .po file for that language and discard the other ones. They might be interesting for Windows users, but not for Linux ones, since translations get converted with iconv into the encoding the user is runnning anyway. And nowadays UTF-8 is the default encoding on Linux. E.g.: if you have zh_TW.po (Big5 encoding), zh_TW.UTF-8.po and zh_TW.cp960.po, please remove zh.po and zh.cp960.po and rename zh_TW.UTF-8.po to zh.po.

• Obsolete no locale: the no locale is obsolete. If you find no.po files, please look into it and find out if it's Norwegian Bokmål or Nynorsk. Usually it's Bokmål. Please rename the file to the proper language code: nb.po for Bokmål, nn.po for Nynorsk. If nb.po and nn.po are already present, remove no.po.

• English US translations: If you have an en_US.po file, please investigate if it really contains translations (i.e. if the source code uses en_GB or another language) and if not, you can safely discard it.

Documentation

TODO:

• Converting xml to PO with xml2po

• Launchpad import

Packages not using the Gettext format

Mozilla

OpenOffice.org

Coding

Individual programming languages

Launchpad integration library

Ubuntu applications use the Launchpad integration library to provide shortcuts to Launchpad to obtain help, report bugs and submit translations. You can see it in action in any Ubuntu application's Help menu:

Here's an example in C of how support for this library was added to Empathy:

=== modified file 'configure.ac'
--- configure.ac        2009-10-13 03:20:58 +0000
+++ configure.ac        2009-10-13 03:26:58 +0000
@@ -131,6 +131,7 @@
    gstreamer-0.10
    unique-1.0
    gnome-keyring-1 >= $KEYRING_REQUIRED
+   launchpad-integration
 ])
 
 PKG_CHECK_MODULES(LIBNOTIFY, libnotify >= $LIBNOTIFY_REQUIRED)

=== modified file 'src/empathy-main-window.c'
--- src/empathy-main-window.c   2009-10-13 03:20:58 +0000
+++ src/empathy-main-window.c   2009-10-13 18:15:41 +0000
@@ -27,6 +27,9 @@
 #include <gtk/gtk.h>
 #include <glib/gi18n.h>
 
+/* Add launchpad hooks */
+#include <launchpad-integration.h>
+
 #include <libempathy/empathy-contact.h>
 #include <libempathy/empathy-utils.h>
 #include <libempathy/empathy-account-manager.h>
@@ -1424,6 +1427,9 @@
 
        main_window_update_status (window, window->account_manager);
 
+       /* Add launchpad hooks */
+       launchpad_integration_add_ui (window->ui_manager, "/menubar/help/LaunchpadItems");
+
        return window->window;
 }
 

=== modified file 'src/empathy-main-window.ui'
--- src/empathy-main-window.ui  2009-10-13 03:19:42 +0000
+++ src/empathy-main-window.ui  2009-10-13 18:17:20 +0000
@@ -243,6 +243,7 @@
         <menu action="help">
           <menuitem action="help_contents"/>
           <menuitem action="help_debug"/>
+          <placeholder name="LaunchpadItems"/>
           <menuitem action="help_about"/>
         </menu>
       </menubar>

Additional documentation

• Gettext documentation

• The GNU libc documentation on Gettext

• GNOME l10n guidelines for developers and the GNOME Translation Project

• Python gettext module documentation

Translation process

TODO

• String freeze
• Translation
• Translation deadline
• Language pack updates

Launchpad

The import queue

All translation templates (.pot files) and translations (.po) end up in the Launchpad Translations import queue before being imported into Launchpad.

Some of the Ubuntu import queues:

• Hardy

• Intrepid

• Jaunty

• Karmic

Templates generally get approved automatically, but sometimes there is some manual work involved in approving them. The Ubuntu Translations Coordinators Team manages the import queue. We've got a mailing list where issues can be discussed and we welcome the help of any community members experienced with translations.

If your project is hosted in Launchpad

General Advice

If you want to make sure that your packages translations are set up correctly, try running:

• For GNOME packages:

cd po; intltool-update -m

and be sure to either fix the issues or report them upstream.

Note: this is only worth trying if the package uses intltool for i18n (i.e. look for intltool.m4 in the top-level source directory)

• For GNU packages:
• For KDE packages:

Sometimes empty .po files caused packages to build from source.

Kubuntu-specific notes

CategoryUbuntuDevelopment CatagoryTranslations