Translations

Dev Week -- Translations for developers -- danilo, dpm, pitti -- Fri Sep 4th, 2009

Summary

(note: some text from the original session has been modified and rearranged in this summary for readability)

Introduction

  • I'm Martin Pitt from the Ubuntu Desktop Team, and more or less the creator of the "language pack" system we have used in Ubuntu since 2005.
  • Hi all, I am Danilo and I lead the Launchpad Translations development team: Launchpad is an open source foundation for Ubuntu i18n and l10n
  • Hi everyone, my name is David Planella, I'm the Ubuntu Translations Coordinator and as such my job is to keep the Ubuntu translation community rocking

In Ubuntu we spend quite some effort on translation of software and move translations around a lot, so that we can clearly separate the actual packaged software from the translations which belong to it, for the following main reasons:

  • Make it as easy as possible for non-technical contributors to help translating software.
  • Deliver translation updates to stable Ubuntu releases without having to touch the actual software packages, and thus jeopardizing their stability.
  • Have a good control which translations land on the release CD images, to mitigate space constraints.

What are language packs?

Langpacks are packages which contain translations for a particular language for software that we ship in Ubuntu main. Universe and multiverse are not currently handled by this system.

The basic idea is that the actual programs are packaged without any translations, and if you are using an e. g. Portuguese desktop, you need to install the Portugese language pack to have Ubuntu talk Portugese instead of English to you.

As a user, you don't usually need to worry about this too much, since the installer takes care to install what you need, though. There's also the "Language selector" in the System menu which allows you to install more. In order to avoid unnecessary downloads, wasted CD space, and wasted installation hard disk space, there is not just one langpack for a particular language, but they are split into "categories" (GNOME, KDE, and common), so that a pure Ubuntu installation does not need to carry GNOME translations.

E. g. the "language-pack-gnome-pt" package ships Portugese translations for all GNOME related packages in main. To further complicate the issue, there is another split between "-base" and "update" packages. The idea is that the bulk of translations is usually ready and done by the final release of Ubuntu, and we want users to not have to download the same things over and over again. So the "-base" packages are big and contain the state of translations as it was at release time, while the "update" packages are small and only contain the delta since the release.

That's why there is not just "language-pack-gnome-pt" (the update package), but also "language-pack-gnome-pt-base". Thus for a single language you usually have a set of six related language-pack-* packages. This makes things a bit convoluted, but makes the system reasonably efficient.

Translation formats

By far the most known and used method of translating Linux software is "gettext". It

  • wraps the translatable strings in the software into a special function _("Hello")

  • extracts those strings into a template file which contains all the transatable strings (called the "PO template")
  • compiles human-readable and editable translation files (*.po) to binary "*.mo" files which provide super-fast access at runtime
  • uses the installed *.mo files at runtime to map an English string to a translated string.

A typical record in a gettext PO file looks like this: 

   msgid "Good morning"
   msgstr "Доброй утро"

(this would be in the ru.po file for Russian)

Launchpad and the Ubuntu langpack system have fully supported gettext from day one.

Unfortunately there is not just gettext in the Linux world, but also other vendor specific systems, mainly due to the fact that these appliciations did not originate in the Unix world.

The prime examples here are Mozilla software (Firefox et al) which use "XPI", and OpenOffice.org which uses a system called "SDF".

Launchpad and langpacks grew support for XPI about a year ago, so that Launchpad can be used to translate Mozilla software now. SDF is not yet handled by Launchpad or langpacks.

For about a week now in karmic, we also started handle GNOME help file translations.

While they use gettext in principle, the translated files are assembled at build time, and packages ship the readily translated XML files and translated screenshots directly.

They take a lot of space, so we now strip them from the actual packages, temporarily park them in Launchpad, and put them into the language packs. But they are really just copied verbatim right now, there is no Launchpad support for updating the help translations yet.

Flow for gettext translations

Since gettext is pretty much the only system which you should need to know, I would like to concentrate on that from now on.

I'd like to explain how translations make their way through this system, to allow developers to be aware of the needs of translators, and how translations make it to the final desktop.

The 1000 m perspective looks like this:

translations in upstream tarball → extract at package build time → import into Launchpad translation community → add/change strings on Launchpad Launchpad translation export → sort them by language and category → generate language-pack-*, and upload them Danilo and David will talk in detail about the Launchpad part later on, so I'll give some details on the packaging related bits.

Build time extraction

The majority of translations come from the already existing shipped PO files in the upstream tarballs. These need to be extracted and imported into Launchpad, and the compiled MO files be removed from built application packages. This is done by a script "pkgstriptranslations" from the "pkgbinarymangler" package. That package is installed in the Ubuntu build servers, but of course you can also install it locally to see what it does.

For the import to actually succeed and work well, packages must ship or build an up-to-date PO template, i. e. the template must be an accurate list of all translatable strings in the application.

It is greatly preferred to have this generated at build time (usually with "intltool-update -p"), to ensure that it isn't stale, and also contains the added/changed strings we do in Ubuntu patches.

If the package uses cdbs and includes gnome.mk or translations.mk, this will be taken care of automatically. All other packages need to be fixed to build a PO template. (This should be the case for almost all packages in Ubuntu main nowadays.)

Package structure

As Martin mentioned, POT and PO files are produced as part of binary builds: for Launchpad to import translations correctly, make sure your builds do produce POT files, or translations will not be up to date (or not imported at all with a new package). Also, note that this process happens only for source packages which are in Ubuntu 'main'. Note that you can have multiple translation templates (POT) for different purposes. Eg. a library POT and main UI POT: but make sure that you keep relevant translation PO files in the same subdirectory as their respective POTs. Also, don't worry about merging PO files with latest POT files: Launchpad does that for you with a very smart algorithm not losing any contributions and worrying about conflicts.

KDE is special in the way POT files are built and where translations are pulled out of: https://wiki.ubuntu.com/Translations/Upstream/KDE/KubuntuTranslationsLifecycle

After all translation files are stripped off, they end up in Launchpad translations import queue: https://translations.launchpad.net/ubuntu/+imports

Import queue

Originally, when they enter the import queue, they are put into Needs review state. For templates, if a base path and filename matches a template Launchpad has previously imported in that source package, it is considered an update of that template, attached to it and marked as Approved.

For translations, Launchpad tries to match them against existing templates and existing language codes. Launchpad on purpose recognizes only "the shortest possible" language codes: use "es.po" and "de.po" and not "es_ES.po" or "de_DE.po".

For anything that can't be automatically approved, it's stays in the queue for someone to look at. If you wonder who that someone might be, I introduce you to...

dpm can tell us more about what UTC stands for Smile :)

UTC stands for Ubuntu Translations Coordinator or the Ubuntu Translations Coordinators Team

The Ubuntu Translations Coordinators team is a group of wonderful people who takes care of all the manual adjustments, reporting more technical issues, etc. and in short caring for Ubuntu translations.

Here you can see them sporting their good looks: https://launchpad.net/~ubuntu-translations-coordinators/+mugshots

The team was born from the intention of making the technical and all the behind-the-scenes work more open to the community. As such Launchpad Translations has been progressively getting more permissions on different levels and granting them to these trusted community members, so they can participate in the process.

One of the main tasks of the UTC team is to manage the imports queue and manually approve, tweak or block translation templates which the auto-approver script cannot automatically handle. They can also decide which translations must be included in language packs.

And here's a link to the Karmic imports queue, for those interested in horribly long links: https://translations.edge.launchpad.net/ubuntu/karmic/+imports?field.filter_extension=pot&field.filter_status=NEEDS_REVIEW&start=0&batch=150

Also, if you want your package (in 'main') translations exported into language packs, you can have UTC team set it up: if they don't, you'll have to manually download translation tarballs from Launchpad and use that export when building updated version of the package.

Translation

After files have been put into 'Approved' state (either automatically or manually), they are imported into Launchpad: usually very quickly, but some uploads can take longer than others (like KDE-l10n and OpenOffice.org with their 20k files each can take around a day).

After POT and PO files have been imported, it's possible to use Launchpad web UI to translate Ubuntu: it provides an easy to use interface but with some advanced features on top of it. The easy way is: go to a web page, look at the English string, fill in a text box with your suggested translation, and save the page.

The more advanced way to do translation is to download a PO file, work on it offline, and then upload it back. And Launchpad will worry about any conflicts, and will do it on per message basis: if you translated the same string someone translated online at the same time, it will make your translation a suggestion and let you know about it.

One of the cool new things is that translators can only work on one version of the project (i.e. trunk series for a project, or Jaunty for Ubuntu), and if relevant, their work will be reflected in all the other versions as well. So, you do a translation of "Open file" in Jaunty. You don't have to go to Karmic to do the translation there as well, it will be automatically propagated. We call this feature "message sharing".

To control access to translation, Launchpad offers translation groups: they are a list of translation teams matched by language. Only people who are part of those translation teams can approve others' translation suggestions: without approval, their translations are never made active.

All this is made possible by the Launchpad development team consisting of henninge, jtv, Ursinha and me: find them in #launchpad and say hi! Also, you can become part of the team as well, remember Launchpad is open source now! Ubuntu has a vibrant translation community as part of 'Ubuntu translators' group. But, I'll let David tell you more about it. I heard to get an Ubuntu translation it doesn't just take a nice platform, you need some people as well? :).

Sure, nice and exciting people!

I'll take it from where Danilo was mentioning translation groups...

First of all, the permissions for translating projects (or distros in the case of Ubuntu) are organised around translation groups, to which project maintainers can assign the translations to.

The biggest translation group in Launchpad is the Ubuntu Translators group. There (https://translations.launchpad.net/+groups/ubuntu-translators) you can see that there is a second level of permission translation communities use to organise themselves: translation groups are containers for translation teams

Translation teams are where the exciting stuff takes place

Communities get organised in teams around Launchpad and use it to translate Ubuntu

At the risk of repeating what Danilo has already said, I'll reinstate that: although everyone with a Launchpad account can provide translation suggestions, only those translators in an Ubuntu translation team will be able to approve them or to submit them themselves.

This means that everyone can get introduced to the world of translations easily, but at the same time, only experienced translators will be able to accept suggestions, maintain a level of translation quality and guide newcomers into the process of becoming full-fledged translators.

Upstream and downstream

Ubuntu makes a lot of use of upstream software. Some of it is Ubuntu's own, like upstart or jockey. And yet others are completely independent like Evolution or Firefox.

With package builds for 'external upstream' applications, you usually get translations from upstream integrated by including the upstream PO files in the build. For 'internal upstream', they are usually hosted in Launchpad as separate projects, and they require some care to make sure people are not confused about where to do their translation.

Note that upstreams usually do not update translations for 'older' versions: that's what Launchpad allows Ubuntu to do. You can still update Hardy translations and they will be reflected in the next language pack update. now, how do we get to language packs?

Exporting language pack tarballs

After translations have been done in Launchpad, Launchpad aggregates all the translations for a single Ubuntu release and puts them in a tarball. Launchpad calls that "language packs", but they are just the base tarballs used to construct final language packs you get installed on your system.

Launchpad does weekly exports of language pack tarballs, with the following schedule: http://dev.launchpad.net/Translations/LanguagePackSchedule

After they are produced, they are listed on distribution release language packs page, eg. https://translations.launchpad.net/ubuntu/jaunty/+language-packs

There are two slightly different types of tarballs Launchpad can produce: either a full tarball (for pitti called "base") containing all translations for templates marked as included in language packs (don't forget about that bit), or only those translations which have been updated since the last full language pack was released — called a "delta language pack" in Launchpad, and "update" package in Ubuntu.

  • QUESTION: how launchpad handles the translation's return to its original mantainer?

  • ANSWER (Danilo): always an interesting matter: Launchpad provides two things: a translation platform for Ubuntu and for projects who use Launchpad as their base translation portal. in case of Ubuntu, translations done in Launchpad are mostly updates to existing upstream translations. With a wide variety of upstreams that Ubuntu uses, there is simply no way Launchpad can know all the ways to send updated translations back in good manners (i.e. not considered spam or aggressive). So, Launchpad provides a facility to help translators submit their work upstream themselves: when they go to export a translation from Launchpad, they can choose to export only those translations which have been changed by ticking the 'Export only changed translations' box on PO file export page. That file can then be sent to the original maintainer for inspection and merging with the upstream translation

I want us to get back to language pack production: I'll let pitti tell you what happens next after Launchpad produces tarballs with translation files

langpack-o-matic

The upstream and Ubuntu community translations get merged together in Launchpad, and then regularly get exported as a huge tarball which contains all translations for all applications.

The job of dissecting this 500 MB monster and producing installable debs is done by a set of scripts that I called "langpack-o-matic".

It has a set of package skeletons for language-pack-*-base and language-pack, and instantiates a group (base/update and gnome/kde/common) of them for each language that is present in the export.

For deciding what is a GNOMEish or a KDEish package it currently uses some heuristics, looking at the package description, dependencies, and so on.

Based on the categorization and language, it sorts the files into the generated language-pack-* source packages. It also adds some extra data, such as converting Mozilla-related gettext translations into XPI files, or ship flag images for KDE.

Testing

For the current Ubuntu development release, langpack-o-matic uploads the generated langpacks straight to the archive, i. e. Karmic at the moment.

That way, they get maximum testing, and we aren't concerned about small regressions within the development release.

For stable releases we need to apply more care; since translations have the potential to break software, or just regress in quality, they need to get thorough testing before they get uploaded to e. g. jaunty-updates.

For this, we have a personal package archive where langpack-o-matic uploads updates for stable Ubuntu releases on a weekly basis. If you are translating Ubuntu software on Launchpad, or just would like to help testing, please enable this PPA to always get the latest translations, and report problems immediately.

Usually, the PPA packages are uploaded to -proposed once a month, then dpm sends out a call for testing on the translators mailing list, and once we can be reasonably sure to not have broken much, they get to -updates for general consumption.

This answers QUESTION: How often the language-pack-update are updated at stable releases?

Q & A

  • QUESTION: A bit late, but does the _() function also works in Python? What module do you need to import?

  • ANSWER (Martin): Python has a gettext module for that. It doesn't export _() by itself, but it's easy enough to do it with 

from gettext import gettext as _

  • QUESTION: if a newbie translator do work at launchpad and the original translator of that app makes work too outside of launchpad , how launchpad handles the collissions?

  • ANSWER (Danilo): so, Launchpad has a "smart" algorithm for deciding what takes precedence. If Launchpad imports an upstream translation, it updates it with every change coming from upstream. Launchpad basically "tracks" the upstream translation. However, if someone modified that translation in Launchpad *on purpose*, we keep it, and mark the newly imported upstream one as "needing review". If someone did a translation in Launchpad which didn't exist upstream, but is later introduced there, we give preference to upstream translation. Basically (re QUESTION), the rule is: only if it was modified on purpose in Launchpad, it takes preference; otherwise upstream translation takes precedence

  • QUESTION: What about projects that have their upstream on Launchpad?

  • ANSWER (Danilo): Projects like these do not have to worry about any integration because everything happens in Launchpad; if they ship translations in Ubuntu, they might be for a different release so that might take manual merge effort for now

  • QUESTION: is there a page in launchpad where I can see all untranslated strings for a specific language so I can just start translating? or do I have to choose a source package first ?

  • ANSWER: There is no such page in Launchpad, though you will get a list of recommendations of what could use some help in translating on your personal page with our 3.0 release coming in ~3 weeks. Note that Launchpad is not only about Ubuntu, though Ubuntu is the big part of it. There is such a page for Ubuntu, eg. look at https://translations.launchpad.net/ubuntu/karmic/+lang/sr. It's a long list, though Smile :)

Thanks all for your attention!

Full transcript

UTC 

(12:00:34 PM) dholbach: dpm, danilos, pitti: the floor is yours!
(12:00:37 PM) pitti: Hello all!
(12:00:39 PM) danilos: dholbach: thanks
(12:00:44 PM) pitti: I'm Martin Pitt from the Ubuntu Desktop Team, and more or less the creator of the "language pack" system we have used in Ubuntu since 2005.
(12:00:51 PM) danilos: Hi all, I am Danilo and I lead the Launchpad Translations development team: Launchpad is an open source foundation for Ubuntu i18n and l10n
(12:01:10 PM) dpm: Hi everyone, my name is David Planella, I'm the Ubuntu Translations Coordinator and as such my job is to keep the Ubuntu translation community rocking
(12:01:27 PM) pitti: In Ubuntu we spend quite some effort on translation of software and move translations around a lot, so that we can clearly separate the actual packaged software from the translations which belong to it, for the following main reasons:
(12:01:42 PM) pitti:  * Make it as easy as possible for non-technical contributors to help translating software.
(12:01:49 PM) pitti:  * Deliver translation updates to stable Ubuntu releases without having to touch the actual software packages, and thus jeopardizing their stability.
(12:01:59 PM) pitti:  * Have a good control which translations land on the release CD images, to mitigate space constraints.
(12:02:12 PM) pitti: == What are language packs? ==
(12:02:20 PM) pitti: Langpacks are packages which contain translations for a particular language for software that we ship in Ubuntu main. Universe and multiverse are not currently handled by this system.
(12:02:29 PM) pitti: The basic idea is that the actual programs are packaged without any translations, and if you are using an e. g. Portugese desktop, you need to install the Portugese language pack to have Ubuntu talk Portugese instead of English to you.
(12:02:51 PM) pitti: As an user, you don't usually need to worry about this too much, since the installer takes care to install what you need, though. There's also the "Language selector" in the System menu which allows you to install more.
(12:03:14 PM) pitti: In order to avoid unnecessary downloads, wasted CD space, and wasted installation hard disk space, there is not just one langpack for a particular language, but they are split into "categories" (GNOME, KDE, and common), so that a pure Ubuntu installation does not need to carry GNOME translations.
(12:03:34 PM) pitti: E. g. the "language-pack-gnome-pt" package ships Portugese translations for all GNOME related packages in main.
(12:03:56 PM) pitti: To further complicate the issue, there is another split between "-base" and "update" packages. The idea is that the bulk of translations is usually ready and done by the final release of Ubuntu, and we want users to not have to download the same things over and over again. So the "-base" packages are big and contain the state of translations as it was at release time, while the "update" packages
(12:03:57 PM) pitti: are small and only contain the delta since the release.
(12:04:19 PM) pitti: That's why there is not just "language-pack-gnome-pt" (the update package), but also "language-pack-gnome-pt-base".
(12:04:33 PM) pitti: Thus for a single language you usually have a set of six related language-pack-* packages. This makes things a bit convoluted, but makes the system reasonably efficient.
(12:04:46 PM) pitti: Questions so far about this split?
(12:05:28 PM) pitti: seems not
(12:05:36 PM) pitti: == Translation formats ==
(12:05:46 PM) pitti: By far the most known and used method of translating Linux software is "gettext".
(12:05:50 PM) pitti: It
(12:05:57 PM) pitti:  * wraps the translatable strings in the software into a special function _("Hello")
(12:06:12 PM) pitti:  * extracts those strings into a template file which contains all the transatable strings (called the "PO template")
(12:06:24 PM) pitti:  * compiles human-readable and editable translation files (*.po) to binary "*.mo" files which provide super-fast access at runtime
(12:06:36 PM) pitti:  * uses the installed *.mo files at runtime to map an English string to a translated string.
(12:06:49 PM) pitti: A typical record in a gettext PO file looks like this:
(12:06:55 PM) pitti:    msgid "Good morning"
(12:06:56 PM) pitti:    msgstr "Доброй утро"
(12:07:10 PM) pitti: (this would be in the ru.po file for Russian)
(12:07:24 PM) pitti: Launchpad and the Ubuntu langpack system have fully supported gettext from day one.
(12:07:42 PM) pitti: Unfortunately there is not just gettext in the Linux world, but also other vendor specific systems, mainly due to the fact that these appliciations did not originate in the Unix world.
(12:07:59 PM) pitti: The prime examples here are Mozilla software (Firefox et al) which use "XPI", and OpenOffice.org which uses a system called "SDF".
(12:08:16 PM) pitti: Launchpad and langpacks grew support for XPI about a year ago, so that Launchpad can be used to translate Mozilla software now. SDF is not yet handled by Launchpad or langpacks.
(12:08:29 PM) pitti: For about a week now in karmic, we also started handle GNOME help file translations.
(12:08:42 PM) pitti: While they use gettext in principle, the translated files are assembled at build time, and packages ship the readily translated XML files and translated screenshots directly.
(12:08:58 PM) pitti: They take a lot of space, so we now strip them from the actual packages, temporarily park them in Launchpad, and put them into the language packs. But they are really just copied verbatim right now, there is no Launchpad support for updating the help translations yet.
(12:09:08 PM) pitti: questions about translation format?
(12:09:18 PM) pitti: please also yell in #chat if I'm too fast/slow
(12:09:51 PM) pitti: ok, so let's talk a little how translations make their way from the translator to the user's desktop
(12:09:56 PM) pitti: == Flow for gettext translations ==
(12:10:22 PM) pitti: Since gettext is pretty much the only system which you should need to know, I would like to concentrate on that from now on.
(12:10:39 PM) pitti: I like to explain how translations make their way through this system, to allow developers to be aware of the needs of translators, and how translations make it to the final desktop.
(12:10:52 PM) pitti: The 1000 m perspective looks like this:
(12:10:59 PM) pitti: (details will follow, don't worry)
(12:11:02 PM) pitti: translations in upstream tarball → extract at package build time → import into Launchpad
(12:11:14 PM) pitti: translation community → add/change strings on Launchpad
(12:11:27 PM) pitti: Launchpad translation export → sort them by language and category → generate language-pack-*, and upload them
(12:11:46 PM) pitti: Danilo and David will talk in detail about the Launchpad part later on, so I'll give some details on the packaging related bits.
(12:12:03 PM) pitti: == build time extraction ==
(12:12:17 PM) pitti: The majority of translations come from the already existing shipped PO files in the upstream tarballs. These need to be extracted and imported into Launchpad, and the compiled MO files be removed from built application packages.
(12:12:36 PM) pitti: This is done by a script "pkgstriptranslations" from the "pkgbinarymangler" package. That package is installed in the Ubuntu build servers, but of course you can also install it locally to see what it does.
(12:12:50 PM) pitti: For the import to actually succeed and work well, packages must ship or build an up-to-date PO template, i. e. the template must be an accurate list of all translatable strings in the application.
(12:13:03 PM) pitti: It is greatly preferred to have this generated at build time (usually with "intltool-update -p"), to ensure that it isn't stale, and also contains the added/changed strings we do in Ubuntu patches.
(12:13:25 PM) pitti: If the package uses cdbs and includes gnome.mk or translations.mk, this will be taken care of automatically. All other packages need to be fixed to build a PO template. (This should be the case for almost all packages in Ubuntu main nowadays.)
(12:13:48 PM) ***pitti hands mike to danilos
(12:14:02 PM) danilos: thanks pitti; so, let me go on a bit with this
(12:14:09 PM) danilos: = Package structure =
(12:14:09 PM) danilos: As Martin mentioned, POT and PO files are produced as part of binary builds: for Launchpad to import translations correctly, make sure your builds do produce POT files, or translations will not be up to date (or not imported at all with a new package).  Also, note that this process happens only for source packages which are in Ubuntu 'main'.
(12:14:29 PM) danilos: Note that you can have multiple translation templates (POT) for different purposes.  Eg. a library POT and main UI POT: but make sure that you keep relevant translation PO files in the same subdirectory as their respective POTs.
(12:14:49 PM) danilos: Also, don't worry about merging PO files with latest POT files: Launchpad does that for you with a very smart algorithm not losing any contributions and worrying about conflicts.
(12:15:09 PM) danilos: KDE is special in the way POT files are built and where translations are pulled out of: https://wiki.ubuntu.com/Translations/Upstream/KDE/KubuntuTranslationsLifecycle
(12:15:30 PM) danilos: After all translation files are stripped off, they end up in Launchpad translations import queue: https://translations.launchpad.net/ubuntu/+imports
(12:15:49 PM) danilos: = Import queue =
(12:15:49 PM) danilos: Originally, when they enter the import queue, they are put into 'Needs review' state.  For templates, if a base path and filename matches a template Launchpad has previously imported in that source package, it is considered an update of that template, attached to it and marked as 'Approved'.
(12:16:10 PM) danilos: For translations, Launchpad tries to match them against existing templates and existing language codes.  Launchpad on purpose recognizes only  "the shortest possible" language codes: use "es.po" and "de.po" and not "es_ES.po" or "de_DE.po".
(12:16:28 PM) danilos: For anything that can't be automatically approved, it's stays in the queue for someone to look at.  If you wonder who that someone might be, I introduce you to...
(12:16:40 PM) danilos: dpm: tell us more about what UTC stands for :)
(12:16:55 PM) ***dpm gets the mike
(12:17:29 PM) dpm: UTC stands for Ubuntu Translations Coordinator or the
(12:17:39 PM) dpm: Ubuntu Translations Coordinators Team
(12:17:57 PM) dpm: The Ubuntu Translations Coordinators team is a group of wonderful people who takes care of all the manual adjustments, reporting more technical issues and in short caring for Ubuntu translations.
(12:18:16 PM) dpm: Here you can see them sporting their good looks: https://launchpad.net/~ubuntu-translations-coordinators/+mugshots
(12:18:39 PM) dpm: The team was born from the intention of making the technical and all the behind-the-scenes work more open to the community. As such Launchpad Translations has been progressively getting more permissions on different levels and granting them to these trusted community members.
(12:19:04 PM) dpm: So they can participate in the process
(12:19:26 PM) dpm: One of the main tasks of the UTC team is to manage the imports queue and manually approve, tweak or block translation templates which the auto-approver script cannot automatically handle. They can also decide which translations must be included in language packs.
(12:20:28 PM) dpm: And here's a link to the Karmic imports queue, for those interested in horribly long links: https://translations.edge.launchpad.net/ubuntu/karmic/+imports?field.filter_extension=pot&field.filter_status=NEEDS_REVIEW&start=0&batch=150
(12:21:30 PM) danilos: dpm: thanks!
(12:21:37 PM) ***danilos fights with dpm over mike
(12:21:46 PM) ***pitti stumbles over the cable
(12:21:48 PM) danilos: Also, if you want your package (in 'main') translations exported into language packs, you can have UTC team set it up: if they don't, you'll have to manually download translation tarballs from Launchpad and use that export when building updated version of the package.
(12:22:18 PM) danilos: Does anyone have any quick questions so far about how stuff gets into the queue and how it gets approved?
(12:22:59 PM) dpm: QUESTION: What is the official way to contact UTC?
(12:23:38 PM) dpm: The UTC has got a mailing list, and you can also contact them by filing a request to the Answers system on the ubuntu-translations project
(12:23:46 PM) dpm: here are the relevant links:
(12:24:18 PM) dpm: https://wiki.ubuntu.com/Translations/Contact/#Ubuntu%20Translations%20Coordination
(12:24:34 PM) dpm: https://answers.edge.launchpad.net/ubuntu-translations/+addquestion
(12:25:02 PM) dpm: following the first link you'll also be able to consult the public mailing list archives
(12:25:18 PM) dpm: Any other questions on UTC? Or anything else so far?
(12:25:59 PM) danilos: I guess not :)
(12:26:05 PM) danilos: so, let's see what happens next
(12:26:09 PM) danilos: = Translation =
(12:26:09 PM) danilos: After files have been put into 'Approved' state (either automatically or manually), they are imported into Launchpad: usually very quickly, but some uploads can take longer than others (like KDE-l10n and OpenOffice.org with their 20k files each can take around a day).
(12:26:31 PM) danilos: After POT and PO files have been imported, it's possible to use Launchpad web UI to translate Ubuntu: it provides an easy to use interface but with some advanced features on top of it.  The easy way is: go to a web page, look at the English string, fill in a text box with your suggested translation, and save the page.
(12:26:52 PM) danilos: The more advanced way to do translation is to download a PO file, work on it offline, and then upload it back.  And Launchpad will worry about any conflicts, and will do it on per message basis: if you translated the same string someone translated online at the same time, it will make your translation a suggestion and let you know about it.
(12:27:13 PM) danilos: One of the cool new things is that translators can only work on one version of the project (i.e. trunk series for a project, or Jaunty for Ubuntu), and if relevant, their work will be reflected in all the other versions as well.
(12:27:29 PM) danilos: So, you do a translation of "Open file" in Jaunty.  You don't have to go to Karmic to do the translation there as well, it will be automatically propagated.  We call this feature "message sharing".
(12:27:45 PM) danilos: To control access to translation, Launchpad offers translation groups: they are a list of translation teams matched by language.  Only people who are part of those translation teams can 'approve' others' translation suggestions: without approval, their translations are never made active.
(12:28:11 PM) danilos: All this is made possible by the Launchpad development team consisting of henninge, jtv, Ursinha and me: find them in #launchpad and say hi!  Also, you can become part of the team as well, remember Launchpad is open source now!
(12:28:29 PM) danilos: Ubuntu has a vibrant translation community as part of 'Ubuntu translators' group.  But, I'll let David tell you more about it.
(12:28:47 PM) danilos: dpm: I heard to get an Ubuntu translation it doesn't just take a nice platform, you need some people as well? :)
(12:29:09 PM) dpm: sure, nice and exciting people
(12:29:44 PM) dpm: I'll take it from where Danilo was mentioning translation groups...
(12:30:16 PM) dpm: First of all, the permissions for translating projects (or distros in the case of Ubuntu) are organised around _translation groups_, to which project maintainers can assign the translations to.
(12:30:39 PM) dpm: The biggest translation group in Launchpad is the Ubuntu Translators group. There (https://translations.edge.launchpad.net/+groups/ubuntu-translators) you can see that there is a second level of permission translation communities use to organise themselves: _translation groups_ are containers for _translation teams_
(12:31:12 PM) dpm: Translation teams are where the exciting stuff takes place
(12:32:05 PM) dpm: Communities get organised in teams around Launchpad and use it to translate Ubuntu
(12:33:10 PM) dpm: At the risk of repeating what Danilo has already said, I'll reinstate that: although everyone with a Launchpad account can provide translation suggestions, only those translators in an Ubuntu translation team will be able to approve them or to submit them themselves.
(12:33:49 PM) dpm: this means that everyone can get introduced to the world of translations easily
(12:34:50 PM) dpm: but at the same time, only experienced translators will be able to accept suggestions, maintain a level of translation quality and guide newcomers into the process of becoming full-fledged translators
(12:35:45 PM) dpm: so let's talk about upstream/downstream relationships
(12:35:53 PM) dpm: danilo, do you want to take it from there?
(12:36:12 PM) danilos: sure, thanks dpm
(12:36:17 PM) danilos: = Upstream and downstream =
(12:36:17 PM) danilos: Ubuntu makes a lot of use of upstream software.  Some of it is Ubuntu's own, like upstart or jockey.  And yet others are completely independent like Evolution or Firefox.
(12:36:43 PM) danilos: With package builds for 'external upstream' applications, you usually get translations from upstream integrated by including the upstream PO files in the build. For 'internal upstream', they are usually hosted in Launchpad as separate projects, and they require some care to make sure people are not confused about where to do their translation.
(12:37:04 PM) danilos: Note that upstreams usually do not update translations for 'older' versions: that's what Launchpad allows Ubuntu to do.  You can still update Hardy translations and they will be reflected in the next language pack update.
(12:37:17 PM) danilos: now, how do we get to language packs?
(12:37:24 PM) danilos: = Exporting language pack tarballs =
(12:37:24 PM) danilos: After translations have been done in Launchpad, Launchpad aggregates all the translations for a single Ubuntu release and puts it in a tarball.  Launchpad calls that "language packs", but they are just the base tarballs used to construct final language packs you get installed on your system.
(12:37:49 PM) danilos: Launchpad does weekly exports of language pack tarballs, with the following schedule: http://dev.launchpad.net/Translations/LanguagePackSchedule
(12:38:12 PM) danilos: After they are produced, they are listed on distribution release language packs page, eg. https://translations.launchpad.net/ubuntu/jaunty/+language-packs
(12:38:35 PM) danilos: There are two slightly different types of tarballs Launchpad can produce: either a full tarball (for pitti called "base") containing all translations for templates marked as included in language packs (don't forget about that bit), or only those translations which have been updated since the last full language pack was released — called a "delta language pack" in Launchpad, and "update" package in Ubuntu.
(12:38:58 PM) danilos: QUESTION: how launchpad handles the translation's return to its original mantainer?
(12:39:42 PM) danilos: always an interesting matter: Launchpad provides two things: a translation platform for Ubuntu and for projects who use Launchpad as their base translation portal
(12:40:05 PM) danilos: in case of Ubuntu, translations done in Launchpad are mostly updates to existing upstream translations
(12:40:50 PM) danilos: With a wide variety of upstreams that Ubuntu uses, there is simply no way Launchpad can know all the ways to send updated translations back in good manners (i.e. not considered spam or aggressive)
(12:41:29 PM) danilos: so, Launchpad provides a facility to help translators submit their work upstream themselves: when they go to export a translation from Launchpad, they can choose to export only those translations which have been changed
(12:41:46 PM) danilos: by ticking the 'Export only changed translations' box on PO file export page
(12:42:10 PM) danilos: that file can then be sent to the original maintainer for inspection and merging with the upstream translation
(12:43:03 PM) danilos: I hope this answers this question
(12:43:29 PM) danilos: I want us to get back to language pack production: I'll let pitti tell you what happens next after Launchpad produces tarballs with translation files
(12:43:44 PM) pitti: == langpack-o-matic ==
(12:43:51 PM) pitti: The upstream and Ubuntu community translations get merged together in Launchpad, and then regularly get exported as a huge tarball which contains all translations for all applications.
(12:44:05 PM) pitti: The job of dissecting this 500 MB monster and producing installable debs is done by a set of scripts that I called "langpack-o-matic".
(12:44:23 PM) pitti: It has a set of package skeletons for language-pack-*-base and language-pack, and instantiates a group (base/update and gnome/kde/common) of them for each language that is present in the export.
(12:44:43 PM) pitti: For deciding what is a GNOMEish or a KDEish package it currently uses some heuristics, looking at the package description, dependencies, and so on.
(12:44:55 PM) pitti: Based on the categorization and language, it sorts the files into the generated language-pack-* source packages. It also adds some extra data, such as converting Mozilla related gettext translations into XPI files, or ship flag images for KDE.
(12:45:12 PM) pitti: == Testing ==
(12:45:23 PM) pitti: For the current Ubuntu development release, langpack-o-matic uploads the generated langpacks straight to the archive, i. e. Karmic at the moment.
(12:45:43 PM) pitti: That way, they get maximum testing, and we aren't concerned about small regressions within the development release
(12:45:50 PM) pitti: For stable releases we need to apply more care; since translations have the potential to break software, or just regress in quality, they need to get thorough testing before they get uploaded to e. g. jaunty-updates.
(12:46:10 PM) pitti: For this, we have a personal package archive where langpack-o-matic uploads updates for stable Ubuntu releases on a weekly basis. If you are translating Ubuntu software on Launchpad, or just would like to help testing, please enable this PPA to always get the latest translations, and report problems immediately.
(12:46:32 PM) pitti: Usually, the PPA packages are uploaded to -proposed once a month, then dpm sends out a call for testing on the translators mailing list, and once we can be reasonably sure to not have broken much, they get to -u
(12:46:33 PM) pitti: pdates for general consumption.
(12:46:55 PM) pitti: this answers: fran_dieguez_| QUESTION: How often the language-pack-update are updated at stable releases?
(12:47:13 PM) pitti: == Links ==
(12:47:19 PM) pitti: Details about all the involved processes: https://wiki.ubuntu.com/Translations/TranslationLifecycle
(12:47:27 PM) pitti: pkgbinarymangler package: https://launchpad.net/ubuntu/+source/pkgbinarymangler
(12:47:41 PM) pitti: langpack-o-matic project, bugs, code: https://launchpad.net/langpack-o-matic
(12:47:51 PM) pitti: Weekly langpack PPA: https://launchpad.net/~ubuntu-langpack/+archive
(12:47:58 PM) pitti: == Q & A ==
(12:48:10 PM) pitti: I propose we go in order of #-chat now
(12:48:17 PM) danilos: pitti: yeah, let's do that
(12:48:35 PM) pitti: qense| QUESTION: A bit late, but does the _() function also works in Python? What module do you need to import?
(12:48:39 PM) pitti: I think I'll take that
(12:48:54 PM) pitti: Python has a gettext module for that
(12:49:07 PM) pitti: it doesn't export _() by itself, but it's easy enough to do it with
(12:49:14 PM) pitti: from gettext import gettext as _
(12:49:53 PM) danilos: == More links ==
(12:49:57 PM) danilos: General i18n info for developers (packaging and coding): https://wiki.ubuntu.com/UbuntuDevelopment/Internationalisation (I'll try to have the page in a readable state by tomorrow)
(12:49:57 PM) danilos: Ubuntu import queue: https://translations.launchpad.net/ubuntu/+imports
(12:49:57 PM) danilos: Current language pack tarball schedule: http://dev.launchpad.net/Translations/LanguagePackSchedule
(12:49:58 PM) danilos: Language pack tarballs: https://translations.launchpad.net/ubuntu/karmic/+language-packs
(12:50:00 PM) danilos: Launchpad documentation: https://help.launchpad.net/Translations
(12:50:02 PM) danilos: and back to...
(12:50:05 PM) danilos: = Q & A =
(12:50:14 PM) pitti: fran_dieguez_| and related: QUESTION: if a newbie translator do work at launchpad and the original translator of that app makes work too outside of launchpad , how launchpad handles the collissions?
(12:50:17 PM) pitti: danilos: ^ ?
(12:50:39 PM) danilos: yeah, let me take that
(12:50:55 PM) danilos: so, Launchpad has a "smart" algorithm for deciding what takes precedence
(12:51:23 PM) danilos: if Launchpad imports an upstream translation, it updates it with every change coming from upstream
(12:51:31 PM) danilos: Launchpad basically "tracks" the upstream translation
(12:52:06 PM) danilos: However, if someone modified that translation in Launchpad *on purpose*, we keep it, and mark the newly imported upstream one as "needing review"
(12:52:34 PM) danilos: If someone did a translation in Launchpad which didn't exist upstream, but is later introduced there, we give preference to upstream translation
(12:53:31 PM) danilos: basically (re QUESTION), the rule is: only if it was modified on purpose in Launchpad, it takes preference; otherwise upstream translation takes precedence
(12:53:42 PM) danilos: I'd like to go back to other earlier question now:
(12:53:44 PM) danilos: QUESTION: What about projects that have their upstream on Launchpad?
(12:54:56 PM) danilos: Projects like these do not have to worry about any integration because everything happens in Launchpad; if they ship translations in Ubuntu, they might be for a different release so that might take manual merge effort for now
(12:55:32 PM) danilos: QUESTION: is there a page in launchpad where I can see all untranslated strings for a specific language so I can just start translating ? or do I have to choose a source package first ?
(12:56:27 PM) danilos: There is no such page in Launchpad, though you will get a list of recommendations of what could use some help in translating on your personal page with our 3.0 release coming in ~3 weeks
(12:56:42 PM) danilos: Note that Launchpad is not only about Ubuntu, though Ubuntu is the big part of it
(12:56:53 PM) danilos: There *is* such a page for Ubuntu, eg. look at
(12:57:20 PM) danilos: https://translations.launchpad.net/ubuntu/karmic/+lang/sr
(12:57:34 PM) danilos: it's a long list, though :)
(12:58:19 PM) pitti: ok, time for one more q
(12:58:25 PM) pitti: ah, no, sorry
(12:58:30 PM) pitti: thanks all for your attention!

CategoryTranslations

MeetingLogs/devweek0909/Translations (last edited 2009-09-09 16:30:59 by 35)