LocalesThatDontSuck

Differences between revisions 4 and 18 (spanning 14 versions)
Revision 4 as of 2005-10-19 23:30:45
Size: 1958
Editor: ca-studio-bsr1o-251
Comment: add launchpad URL
Revision 18 as of 2008-08-06 16:38:19
Size: 5919
Editor: localhost
Comment: converted to 1.6 markup
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
 * Created: [[Date(2005-09-26T18:35:38Z)]] by JeffBailey
 * Priority: NeedsPriorit
y
 * People: JeffBailey, JordiMallach
 * Created: <<Date(2005-09-26T18:35:38Z)>> by JeffBailey
 * People: JeffBailey, JordiMallach, MartinPitt
Line 7: Line 6:
 * Interested: MartinPitt
 * Status: UbzSpecification, BrainDump (then DraftSpecification then EditedSpecification then ApprovedSpecification), DistroSpecification
Line 10: Line 7:
 * Branch:
 * Malone bug:
 * Packages affected: locales, langpacks
 * Depends:
 * Dependents:
 [[FullSearch()]]
 * BoF sessions: none yet
 * Packages affected: locales, langpacks, belocs-locales-data
Line 20: Line 12:
In order to server our user communities better, we need to be more responsive and proactive with localisation updates.  There are two solutions proposed here: In order to serve our user communities better, we need to be more responsive and proactive with localisation updates.
Line 22: Line 14:
1) Switch to using the belochs locales packages.

2) Use the belochs locales packages to seed Rosetta, and generate all localisation information from Rosetta.
Line 30: Line 19:
In addition, updating locales requires a complete rebuild of the glibc package. This is a very time and resource intensive build just for the sake of arch-indep files.
Line 34: Line 24:
* Agloolik lives in Iqaluit, Nunavut, Canada and has been using Rosetta to do translations, but wants his gnome calendar to correctly use Sunday as the first day of the week.  * A user from an unsupported locale would like to submit his entry. We accept it via Rosetta, and it gets applied during the next lang-pack build.

 * New locales need to be integrated. The locales maintainer merges with lang-pack, and avoids having to force a rebuild of glibc.
Line 38: Line 30:
* Locales / Belocs-locales packaging  * Locales / Belocs-locales-data(universe) packaging
Line 40: Line 32:
* Langpacks

* Rosetta
 * Langpacks: postinst/postrm scripts, additional content (ship locale data)
Line 46: Line 36:
See also BreezyFirefoxStartPageTranslation.  * Currently locales come from glibc proper. Implementation will move locale data into lang-packs for easier maintenance, and to allow us to move to belocs-locales, which will provide more up-to-date data.

 * Factor out code from the language pack maintainer scripts to a common package `locales`.
Line 50: Line 42:
=== Code === === Glibc package ===

 * All locale data needs to be removed from the current locales package, generated by the glibc build.

 * The locale binary needs to be moved to the locales package.

 * The locale-gen script from the glibc locales package needs to be updated to make use of the new /var/lib/locales/supported.d/<lang> files. It should have two modes. When called with no arguments, it should generate all locales listed in the files in /var/lib/locales/supported.d/*. If called with one argument (a lang name), it should generate all locales listed in /var/lib/locales/supported.d/<lang>

 * Import changes from belocs-locales-bin into glibc. The patches from belocs-locales-bin are required in order to generate locales from belocs-locales-data. These changes would be pushed upstream as needed.

 * Completely remove the currently existing debconf question, since it is unnecessary. Locales get installed and removed automatically.

 * The locales package would continue to ship a full SUPPORTED file for backwards compatibility with packages that use it. Packages that use this file would need to be updated in order to make use of the better mechanism of /var/lib/locales/supported.d/
  * ColinWatson: This will be kind of interesting. `localechooser` uses SUPPORTED to work out what locales to support in the installer; furthermore, it uses `localedef` on a number of those locales in order to be able to sort countries by the collation order appropriate to the language, and thus would end up build-depending on a huge pile of language packs under this scheme (which would suck). We ''could'' remove the nice sorting, but it seems a real shame since it works well ...

=== language-packs packages ===

 * Move the belocs-locales data into language packs build.

 * language packs ship a file /var/lib/locales/supported.d/<lang> (which is a subset of the former /usr/share/i18n/SUPPORTED file).

 * The common postinst script calls `install-language-locales` ''lang-code''. The common postrm script calls `remove-language-locales` ''lang-code''. These scripts need to be written and provided by a new lang-pack-base package that all lang-packs would depend on. Initially these scripts would only need to call locale-gen with the lang name for install, and then insure removal of generated files on remove.

 * Lang-packs would include all of the proper locales for that language (e.g. en_* for English).

=== langpack-o-matic ===

 * langpack-o-matic already provides the mechanics for shipping additional data in the language packs. This will be used to ship the locale data.

=== belocs-locales-data and belocs-locales-bin ===

These packages would no longer be needed once relevant parts are merged into lang-packs and glibc.
Line 54: Line 77:
== Outstanding issues == Belocs-locales seems to want to be a superset of glibc locales, but it appears to be missing several locales that are in glibc proper. If anything is missing from belocs, then the missing part needs to be investigated for inclusion as well. Those missing from belocs are very probably no longer needed for various, justified reasons (countries changing name, ISO 639 code updates...).

Specifically, several locales in Breezy were marked to be removed after the "Sarge" release. These should not be carried forward.

 * Release Notes: Point out that the user has to manually fix his locales when he manually added locales without the corresponding language packs installed.
Line 57: Line 84:

Attendees generally agreed that having locales in glibc was sub-optimal, and that not being able to get locales merged upstream was just an added difficulty.

Consensus was that moving to lang-packs now would also enable us move maint of the locale data to Rosetta at a later date (hopefully, dapper+1).

A new spec needs to be created for Rosetta, to plan for the move of locales data.

----
CategorySpec

Summary

In order to serve our user communities better, we need to be more responsive and proactive with localisation updates.

Rationale

Getting locale changes into upstream glibc is unnecessarily hard. The glibc maintainer (correctly) requires proof of the correctness of a change before it's included, but the onus of demonstrating that change falls to the glibc package maintainers. The package maintainers are frequently not qualified to provide this proof, and cannot answer upstreams questions to the needed degree of satisfaction. The glibc upstream maintainer also has a well deserved reputation for being difficult to approach with changes.

In addition, updating locales requires a complete rebuild of the glibc package. This is a very time and resource intensive build just for the sake of arch-indep files.

Use cases

  • A user from an unsupported locale would like to submit his entry. We accept it via Rosetta, and it gets applied during the next lang-pack build.
  • New locales need to be integrated. The locales maintainer merges with lang-pack, and avoids having to force a rebuild of glibc.

Scope

  • Locales / Belocs-locales-data(universe) packaging
  • Langpacks: postinst/postrm scripts, additional content (ship locale data)

Design

  • Currently locales come from glibc proper. Implementation will move locale data into lang-packs for easier maintenance, and to allow us to move to belocs-locales, which will provide more up-to-date data.
  • Factor out code from the language pack maintainer scripts to a common package locales.

Implementation

Glibc package

  • All locale data needs to be removed from the current locales package, generated by the glibc build.
  • The locale binary needs to be moved to the locales package.
  • The locale-gen script from the glibc locales package needs to be updated to make use of the new /var/lib/locales/supported.d/<lang> files. It should have two modes. When called with no arguments, it should generate all locales listed in the files in /var/lib/locales/supported.d/*. If called with one argument (a lang name), it should generate all locales listed in /var/lib/locales/supported.d/<lang>

  • Import changes from belocs-locales-bin into glibc. The patches from belocs-locales-bin are required in order to generate locales from belocs-locales-data. These changes would be pushed upstream as needed.
  • Completely remove the currently existing debconf question, since it is unnecessary. Locales get installed and removed automatically.
  • The locales package would continue to ship a full SUPPORTED file for backwards compatibility with packages that use it. Packages that use this file would need to be updated in order to make use of the better mechanism of /var/lib/locales/supported.d/
    • ColinWatson: This will be kind of interesting. localechooser uses SUPPORTED to work out what locales to support in the installer; furthermore, it uses localedef on a number of those locales in order to be able to sort countries by the collation order appropriate to the language, and thus would end up build-depending on a huge pile of language packs under this scheme (which would suck). We could remove the nice sorting, but it seems a real shame since it works well ...

language-packs packages

  • Move the belocs-locales data into language packs build.
  • language packs ship a file /var/lib/locales/supported.d/<lang> (which is a subset of the former /usr/share/i18n/SUPPORTED file).

  • The common postinst script calls install-language-locales lang-code. The common postrm script calls remove-language-locales lang-code. These scripts need to be written and provided by a new lang-pack-base package that all lang-packs would depend on. Initially these scripts would only need to call locale-gen with the lang name for install, and then insure removal of generated files on remove.

  • Lang-packs would include all of the proper locales for that language (e.g. en_* for English).

langpack-o-matic

  • langpack-o-matic already provides the mechanics for shipping additional data in the language packs. This will be used to ship the locale data.

belocs-locales-data and belocs-locales-bin

These packages would no longer be needed once relevant parts are merged into lang-packs and glibc.

Data preservation and migration

Belocs-locales seems to want to be a superset of glibc locales, but it appears to be missing several locales that are in glibc proper. If anything is missing from belocs, then the missing part needs to be investigated for inclusion as well. Those missing from belocs are very probably no longer needed for various, justified reasons (countries changing name, ISO 639 code updates...).

Specifically, several locales in Breezy were marked to be removed after the "Sarge" release. These should not be carried forward.

  • Release Notes: Point out that the user has to manually fix his locales when he manually added locales without the corresponding language packs installed.

BoF agenda and discussion

Attendees generally agreed that having locales in glibc was sub-optimal, and that not being able to get locales merged upstream was just an added difficulty.

Consensus was that moving to lang-packs now would also enable us move maint of the locale data to Rosetta at a later date (hopefully, dapper+1).

A new spec needs to be created for Rosetta, to plan for the move of locales data.


CategorySpec

LocalesThatDontSuck (last edited 2008-08-06 16:38:19 by localhost)