Ubuntu Wiki

Ubuntu Opportunistic Developers Week March 2010 - Hot rodding your app for translations support - David Planella - Mar 4 2010

(12:01:54 PM) dpm: Welcome to today's first session of Ubuntu Opportunistic Developer Week
(12:02:34 PM) dpm: let's just wait a couple of minutes, to see that everyone gets here...
(12:03:15 PM) dpm: I should also remind you that today is a snippets day
(12:03:37 PM) dpm: of which you'll find all details here: https://wiki.ubuntu.com/PythonSnippets
(12:04:21 PM) dpm: There'll also be a snippets party in #ubuntu-app-devel later, so be sure to join the fun!
(12:04:30 PM) dpm: Anyway, without further ado...
(12:04:48 PM) dpm: [SLIDE 1]
(12:05:29 PM) dpm: Welcome to this session on Hot rodding your application for translations support
(12:06:08 PM) dpm: In the next hour we'll talk on how to get your app ready to speak a multitude of languages
(12:06:39 PM) dpm: and set up so that our awesome translation teams can do their work
(12:06:58 PM) dpm: For those who are not using Lernid, here's a direct link to the slides in this session: http://people.ubuntu.com/~dpm/Quickly-i18n.pdf
(12:07:50 PM) dpm: This talk will be based on quickly as the framework to opportunistically create your application in no time
(12:08:24 PM) dpm: parts of the talk were contributed by didrocks, Didier Roche, who's the main quickly developer
(12:08:50 PM) dpm: Let's start having a quick look at the main players involved in the internationalization game:
(12:09:04 PM) dpm: [SLIDE 2]
(12:09:06 PM) dpm: == GNU Gettext ==
(12:09:30 PM) dpm: Gettext is the underlying and most widely used technology to enable translations of Open Source projects.
(12:09:42 PM) dpm: It defines a standard format of translation files translators can do their work with (PO files)
(12:09:52 PM) dpm: and lets applications load those translations compiled in a binary format (MO files) at runtime.
(12:10:19 PM) dpm: It has implementations for many programming languages, and amongst them, of course, Python.
(12:10:34 PM) dpm: You'll find that the comprehensive gettext manual at http://www.gnu.org/software/gettext/manual/gettext.html can be a very useful reference,
(12:10:54 PM) dpm: The Python implementation of the gettext API is what we'll use to internationalize our project with Quickly today.
(12:11:12 PM) dpm: Needless to say, it also comes with some nifty documentation at http://docs.python.org/library/gettext.html
(12:11:25 PM) dpm: == intltool ==
(12:11:39 PM) dpm: Intltool is a higher level tool that adds functionality to gettext by allowing the extraction of translatable strings from a variety of file formats
(12:11:54 PM) dpm: It has also become a standard tool when implementing internationalization for OSS projects. Nearly all (if not all) GNOME projects, for example, use intltool.
(12:12:06 PM) dpm: == python-distutils-extra ==
(12:12:24 PM) dpm: Python-distutils-extra is a python package that makes it easy to integrate themable icons, documentation and gettext based translations in your python install and build tools, and it's basically an enhancement to python-distutils.
(12:12:38 PM) dpm: The project's page is at http://www.glatzor.de/projects/python-distutils-extra/
(12:12:52 PM) dpm: The three above technologies (gettext, intltool, python-distutils-extra) are transparently used by quickly, so we won't get into much more detail for now.
(12:13:13 PM) dpm: There are also more aspects involved in internationalizing applications, such as font rendering, input methods, etc., but this should get you started for now.
(12:13:37 PM) dpm: == Quickly ==
(12:14:04 PM) dpm: I'll be very brief here and let you figure out more on quickly as we go along
(12:14:24 PM) dpm: For now, it will suffice give you a teaser and tell you that it is the tool which brings back the fun in writing applications!
(12:14:42 PM) dpm: == Launchpad Translations ==
(12:15:14 PM) dpm: Launchpad Translations ( https://translations.launchpad.net/ ) is the collaborative online tool which allows translation communities to be brought together and translate applications online through its web UI.
(12:15:31 PM) dpm: Apart from the very polished UI to provide translations, it has other nice features such as message sharing across project series (translate one message in a series and it instantly propagates to all other shared series),
(12:15:55 PM) dpm: global suggestions (suggestions of translations across _all_ projects in Launchpad), automatic imports of translations and automatic commits to bzr branches, several levels of permissions, and a huge translator base.
(12:16:25 PM) dpm: On the right hand side of the URL I gave you you can see that there are quite a lot of projects using Launchpad to make translations easy both for developers and translators.
(12:16:59 PM) dpm: Ok, enough theory, let's have a go at using quickly to create your first internationalized application
(12:17:47 PM) dpm: You can install Quickly on Karmic or Lucid simply by executing 'sudo apt-get install quickly'
(12:18:34 PM) dpm: or if you are brave, you can try the trunk version (bzr branch lp:quickly)
(12:18:44 PM) dpm: let's focus on the stable version, though
(12:19:20 PM) dpm: [SLIDE 3]
(12:19:41 PM) dpm: You should see an overview of quickly here
(12:20:07 PM) dpm: Quickly has two parts: the core, which basically parses your input and templates.
(12:20:31 PM) dpm: Templates are sets of commands and code generators that are designed to work together in an end to end fashion to help developers write a certain kind of application.
(12:21:00 PM) dpm: with templates, you can then create application or document set.
(12:21:24 PM) dpm: We'll focus there only on i18n with the first template that Quickly 0.2 provides: ubuntu-project (renamed ubuntu-application on the coming 0.4 release!)
(12:21:36 PM) dpm: (this template is using Glade, couchdb, has some nice trick for gedit, use bzr, and complete integration with LaunchPad and debian packaging)
(12:22:00 PM) dpm: For instance, Lernid that some of you may be using, was created with the ubuntu-project template. (http://www.jonobacon.org/?p=2258 for a shot of the story)
(12:22:12 PM) dpm: [SLIDE 4]
(12:22:28 PM) dpm: So, let's create first a simple ubuntu-project (assuming you are using Quickly 0.2, the stable version):
(12:22:42 PM) dpm: You can run this:
(12:22:49 PM) dpm:     quickly create ubuntu-project fooby
(12:23:04 PM) dpm: this tells Quickly to use the ubuntu-project template, and to call what is created "fooby"
(12:23:16 PM) dpm: This causes a bunch of info to be dumped to the command line, but ends with the application being run
(12:23:26 PM) dpm: What Quickly did was to copy over basically a sample application, and do some text switcheroos to customize the app
(12:23:35 PM) dpm: You can see there the ui which contains some text that needs translation.
(12:23:55 PM) dpm: To start making change to your app, cd to it (normally,
(12:24:00 PM) dpm:     cd fooby
(12:24:21 PM) dpm: You can then edit your code with $ quickly edit, change the UI with $ quickly glade, and try your changes with $ quickly run
(12:24:38 PM) dpm: You can save your change with $ quickly save
(12:24:45 PM) dpm: changes
(12:25:09 PM) dpm: Finally, to package, share, release your apps so that other will be, with the following commands (not all are necessary): $ quickly package / $ quickly share / $ quickly release
(12:25:51 PM) dpm: I won't cover in any more detail Quickly or the ubuntu-project template here (quickly help <command> does this for you),
(12:26:05 PM) dpm: you can find more info at: https://wiki.ubuntu.com/MeetingLogs/devweek0909/QuicklyFun
(12:26:44 PM) dpm: Let's see what you can do in "quickly edit" and "quickly glade": adding internalization support to your app
(12:26:57 PM) dpm: As the project stands now, it has the infrastructure for internationalization in place, but we have to initialize it to enable it.
(12:27:14 PM) dpm: This will include:
(12:27:25 PM) dpm:  * Initializing gettext
(12:27:37 PM) dpm:  * Marking strings for translation
(12:27:48 PM) dpm:  * Updating the translation template
(12:28:02 PM) dpm: [SLIDE 5]
(12:28:06 PM) dpm: First of all, we'll initialize gettext, which will basically be adding four lines of code. Here's how it goes:
(12:28:12 PM) dpm:     quickly edit
(12:28:22 PM) dpm: This will open all your project files in a text editor (Gedit by default)
(12:28:33 PM) dpm: 1. Go to the fooby file and add the following two lines below 'import gtk', near the top of the file:
(12:28:57 PM) dpm:     import gettext
(12:28:57 PM) dpm:     import locale
(12:29:09 PM) dpm: This will import the required modules for internationalization
(12:29:21 PM) dpm: 2. Still on the fooby file, add the following line below the 'import logging, optparse' one, near the end of the file:
(12:29:34 PM) dpm:     gettext.install('fooby', unicode=True)
(12:29:48 PM) dpm: This will install the _() function to mark (and call) translations as such in Python's builtins namespace, based on the 'fooby' domain. The domain basically tells gettext where to load translations from.
(12:30:09 PM) dpm: This will also save you to include 'import gettext' statements in all of your project files using gettext. This will do for your first application
(12:30:25 PM) dpm: Refer to the gettext documentation to find out more about translation domains.
(12:30:34 PM) dpm: 3. Finally, add the following line to 'fooby', before the 'builder = gtk.Builder()' line
(12:30:46 PM) dpm:     locale.textdomain('fooby')
(12:30:57 PM) dpm: This will tell GtkBuilder about the translation domain as well
(12:31:11 PM) dpm: So that was it! Let's move on to marking strings for translation:
(12:31:21 PM) dpm: First of all we'll tackle the .destop file
(12:31:32 PM) dpm: Open the fooby.desktop.in file and prepend the Name and comment fields with an underscore (_), so they look like:
(12:31:50 PM) dpm:     _Name=Fooby
(12:31:50 PM) dpm:     _Comment=Fooby application
(12:32:05 PM) dpm: This will tell intltool that this strings contain translations
(12:32:22 PM) dpm: Next comes the UI. Let's see how you can mark strings in the UI for translation. Try:
(12:32:32 PM) dpm:     quickly glade
(12:32:45 PM) dpm: This will open your UI files in the glade editor.
(12:32:57 PM) dpm: Once opened, click on the "Your application has been created! ..." label, find it in the General > Label field on the right, click on the ellipsis (the three dots) button and...
(12:33:05 PM) dpm: amaze at the fact that it has already been marked as "Translatable", so you won't have to do anything.
(12:33:23 PM) dpm: Right, so next comes something very important that you'll have to bear in mind for all strings you'd like to be translatable in your application:
(12:33:34 PM) dpm:  * using the _() function call.
(12:33:48 PM) dpm: This will mark them as translatable and call gettext to load the translations, and should be used for all messages you'd like to present to users.
(12:34:02 PM) dpm: Let's just see how we can do this.
(12:34:15 PM) dpm: Go back to your fooby file and find the "parser.add_option("-v", "--verbose", action="store_true", dest="verbose", help="Show debug messages")" line near the bottom.
(12:34:34 PM) dpm: We want the "Show debug messages" message to get shown to users in their language, so we'll enclose it with the _() function, and it will look like:
(12:34:54 PM) dpm:     parser.add_option("-v", "--verbose", action="store_true", dest="verbose", help=_("Show debug messages"))
(12:35:06 PM) dpm: Now we're done
(12:35:16 PM) dpm: The last part will be to update the translations template.
(12:35:28 PM) dpm: A translations template is a formatted text file generally named yourproject.pot
(12:35:45 PM) dpm: which contains your project's translatable strings in English and is what translators use as a basis for their translations.
(12:36:04 PM) dpm: You should do this at least before each release, so that translations are put in this template and are up-to-date for translators to work on.
(12:36:22 PM) dpm: It is considered good practice to announce a string freeze (that is, the period in which strings are considered to be stable) a week or two before the release, so that translators know when they can start their work.
(12:36:38 PM) dpm: This can be done in several ways with quickly, let's pick one:
(12:36:54 PM) dpm:     quickly package
(12:37:01 PM) dpm: After running this, you'll notice that (apart from having your application packaged!) there is a 'po' folder containing the translation template, ready for translators to work on.
(12:37:24 PM) dpm: If you open it, you'll notice the format and will see that all the strings you marked for translation are there.
(12:37:39 PM) dpm: You can also do this with 'quickly share', 'quickly release' or directly using the python-distutils-extra command: './setup.py build_i18n'
(12:37:55 PM) dpm: == Launchpad Translations ==
(12:38:28 PM) dpm: If this weren't awesome enough, once you've created a project in Launchpad ('quickly share', 'quickly release' or https://help.launchpad.net/ will help you on that) you can expose it for the world to translate
(12:38:51 PM) dpm: so that you as a developer can use the automatic bzr import/export features to basically "forget" about translations and translators can use the web UI to translate.
(12:39:07 PM) dpm: Let me tell you a bit more on those:
(12:39:11 PM) dpm: [SLIDE 6]
(12:39:20 PM) dpm: == Automatic imports ==
(12:39:33 PM) dpm: Enabling this feature will allow you to automatically import the translation template for your application into Launchpad upon commit, with no further steps required.
(12:39:49 PM) dpm: So the basic workflow will be: hack, hack, hack, update template, commit, have translators automagically see the new strings in Launchpad.
(12:40:15 PM) dpm: You can find more about this at http://blog.launchpad.net/translations/import-translation-templates-from-your-projects-bazaar-branches and http://blog.launchpad.net/general/trying-out-launchpad-translations
(12:40:33 PM) dpm: or at http://blog.launchpad.net/translations/screencast-importing-translation-templates-from-a-bazaar-branch
(12:40:46 PM) dpm: == Automatic exports ==
(12:41:11 PM) dpm: With automatic exports, you'll be able to complete the whole circle for automation: getting translations committed automatically (daily) to a bzr branch of your choice, so that neither you nor translators have to worry to get translations into your project.
(12:41:25 PM) dpm: I personally find this one of the most coolest features
(12:41:37 PM) dpm: Here's more info: http://blog.launchpad.net/general/exporting-translations-to-a-bazaar-branch
(12:41:53 PM) dpm: And here's a screencast on how to enable it http://blog.launchpad.net/translations/screencast-exporting-translations-to-a-bazaar-branch
(12:42:20 PM) dpm: == Permissions ==
(12:42:32 PM) dpm: One very important aspect is how you want translations permissions for your project to be. This basically means choosing who will be responsible for submitting and reviewing those
(12:42:45 PM) dpm: translations for each language.
(12:42:55 PM) dpm: Launchpad is flexible in allowing different levels of openness for translating your project.
(12:43:09 PM) dpm: This generally means that you as a maintainer will have to make a decision to balance openness (open translations for everyone) with quality control (a more closed process with reviewers and a QA workflow).
(12:43:22 PM) dpm: The Launchpad help page on permissions at https://help.launchpad.net/Translations/YourProject/PermissionPolicies explains very well the different permissions you can use (Open, Structured, Restricted and Closed).
(12:43:37 PM) dpm: If you decide for quality, you'll next have to choose to whom you assign the translation of your project.
(12:43:48 PM) dpm: Here is where translation groups come to the rescue.
(12:44:07 PM) dpm: Translation groups are confederations of translation teams, one for each language you can assign as a pack to translate your project.
(12:44:22 PM) dpm: The teams in those groups are considered to be trusted to have experience with translations and generally have a review process in place.
(12:44:33 PM) dpm: Here's a list of all current translation groups: https://translations.launchpad.net/+groups
(12:44:45 PM) dpm: You can see that the two biggest ones are Launchpad Translators and Ubuntu Translators.
(12:45:04 PM) dpm: While you can create a translation group specific to your project, we generally encourage maintainers to choose one of the existing ones,
(12:45:21 PM) dpm: in order to reuse the pool of translators and not to further fragment translations communities.
(12:45:32 PM) dpm: I personally recommend choosing Restricted (or Structured), assigned to the trusted Launchpad Translators or Ubuntu Translators (if your project is Ubuntu-specific) translations groups
(12:45:51 PM) dpm: [SLIDE 7]
(12:46:29 PM) dpm: I'll rapidly go through quickly's incoming features and then we can do some Q&A
(12:46:49 PM) dpm: Quickly 0.4 will bring a lot of new experiences and commodities to the users (more than 200 commits, 6 months of hard work!) and will be delivered in Lucid.
(12:47:05 PM) dpm: Regarding internationalization, all the tedious job of importing/initalizing gettext and adding _() will be done for you in all newly created apps.
(12:47:20 PM) dpm: "$ quickly add pythonfile" will also add one boiler plate file containing that for you.
(12:47:55 PM) dpm: Quickly 0.6 later on will try to achieve the automatic imports and exports previously described on each $ quickly share / $ quickly release command.
(12:48:12 PM) dpm: So, normally, you won't have to bother anymore about localization and being in sync with the awesome work of your contributors
(12:48:24 PM) dpm: just mark strings with _()
(12:48:32 PM) dpm: So that was it!
(12:48:37 PM) dpm: Questions?
(12:51:08 PM) dpm: <gnunezr> QUESTION: How does one get the translation files for each language from the .pot file?
(12:51:21 PM) dpm: That's usually the work of translators
(12:51:49 PM) dpm: when you've committed the .pot file to your bzr branch, and it is exposed in Launchpad with automatic imports
(12:52:15 PM) dpm: translators will see the translatable strings in Launchpad and start doing their work
(12:53:13 PM) dpm: if you've got automatic exports activated, the PO files will be created for you whenever there is a new translation, and committed to your branch of choice
(12:53:48 PM) dpm: So, in short, as a maintainer this will happen automagically for you :)
(12:54:27 PM) dpm: keep them coming :)
(12:55:23 PM) dpm: O I see, gnunerz is asking:
(12:55:35 PM) dpm: QUESTION: I get that... I'm just curious as to how does the manual process goes :)
(12:56:29 PM) dpm: if you were not using Launchpad, you'd announce that there is a new POT file available, and translators would fetch it, create a PO file containing the translation for their language, and would send it back to you
(12:57:00 PM) dpm: The same process: translators always take care of translations, but with a bit more manual work for all people involved
(12:57:16 PM) dpm: So, in shourt, I'd recommend using Launchpad and automatic import/export
(12:57:42 PM) dpm: <alucardni> QUESTION: It is possible to define which languages I want my app be translated to?
(12:58:22 PM) dpm: No, in general you want your application to be available in as many languages as possible, and Launchpad allows translating in almost any language
(12:58:46 PM) dpm: You might be able to limit which translations are build in your project's build infrastructure
(12:59:03 PM) dpm: But you'll have to explicitly set it up to use just some of the translations
(12:59:40 PM) dpm: <gnunezr> QUESTION: Does this process also take care of locale stuff, like how is the date presented to the users?
(12:59:53 PM) dpm: in general, yes
(01:00:13 PM) dpm: but we'd need another session to go into the details :)
(01:00:32 PM) dpm: basically, the app must set up the locale settings right on initialization