2004-12-11

#Ubuntu-Meeting

Ubuntu-Meeting Log 2004-11-??

Topic:Technical Board Meeting - Tuesday 30 November 2004 at 16:00 UTC
Agenda: https://www.ubuntulinux.org/wiki/TechnicalBoardAgenda

pitti   so I copied the ascii file to http://people.ubuntu.com/~pitti/langpacks.txt

pitti   Hi mdz
05:01   mdz     pitti: your agenda item was masked by a markup error, I didn't see it until I tried to edit the page
pitti   mdz: huh?
pitti   mdz: it was fine for me
05:01   mdz http://www.ubuntulinux.org/wiki/TechnicalBoardAgenda
05:02   mdz     elmo,fabbione,kamion,keybuk,lamont,mvo_,sabdfl
05:03   fabb1one        test
05:03   fabb1one        ok much better
05:03   Keybuk  quite finished? ;o)
05:04   sabdfl  hi all
05:04   Keybuk  btw, I'd just like to say, for the record, that Cadbury's are evil
05:04   mdz     ok, we seem to have a quorum
05:04   pitti   Keybuk: what's Cadbury?
05:04   elmo    lol
05:05   Keybuk  pitti: they make chocolate
05:05   elmo    pitti: chocolate manufacturer
05:05   sabdfl  pitti: it's a backup weapon to the jellybean
05:05   mdz     the only predetermined agenda item is: language packs
pitti   understands :-)
05:05   pitti   mdz: that was me, BTW
05:05   Keybuk  (and then feel ill)
05:05   Keybuk  :p
05:05   mdz     if I'm not mistaken, this is carried over from the last TB meeting :-)
05:05   sabdfl  mdz: we had a long discussion during your break
05:05   mdz     ...which I have only skimmed
05:05   pitti   mdz: yes, I distilled the discussion to http://people.ubuntu.com/~pitti/langpacks.txt and to ubuntu-devel
05:06   pitti   however, the discussion on the ml wasn't very active
05:06   pitti   so I would like to talk about it here
05:06   sabdfl  i have a tech question for pitti
05:06   pitti   we don't need to agree about every single detail
05:07   sabdfl  if we set the default build process for packages so that english is installed by default by the package
05:07   pitti   but we should agree on the general architecture
05:07   pitti   sabdfl: this is the default, btw
05:07   sabdfl  and had a mechanism so that a local build could install it's own translations for specific locally required languages, overriding the lang pack
05:07   sabdfl  would this resolve the conflicts issue?
05:07   sabdfl  largely
05:08   pitti   sabdfl: you mean if you locally rebuild the debs from the source package?
05:08   sabdfl  IOW i imagine someone building the package locally would automatically get (a) english and (b) the languages that their Ubuntu langpacks are installed for
05:08   sabdfl  yes
05:08   sabdfl  without having to rebuild the langpack
05:08   pitti   it should be easy to confine the build to some languages, yes
05:08   Keybuk  package would conflict with langpack in that situation, no?
05:08   Kamion  isn't it quite important that local builds = our builds by default?
05:09   pitti   this essentially means to delete all other languages from /usr/share/locale
05:09   mdz     Kamion: yes
05:09   fabbione        Kamion: yes
05:09   sabdfl  Kamion: they would, in the sense that the deb would include english by default, as would ours
05:09   pitti   but this would have similar drawbacks as F5
05:09   Kamion  sabdfl: I mean in all senses, i.e. same list of files
05:09   mdz     if we're going to do language packs, the language pack should become the central point for translation activity
05:09   Kamion  that's really important for developers testing stuff
05:09   pitti   I really thought over this issue
05:10   pitti   IMHO putting the translations into proper debs leads to nothing but trouble
05:10   sabdfl  agreed
05:10   pitti   either our archive size or our mirrors explode
05:10   pitti   and we become totally incompatible to the rest of the deb-world
05:10   mdz     pitti: what do you mean by proper debs?
05:10   mdz     e.g., language packs?
05:11   mdz     I think there are certainly benefits
05:11   pitti   mdz: putting the po files into a real deb (as in the F2 options)
05:11   Kamion  pitti: nod - the approach of grabbing stuff straight to /usr/share/locale/ is looking kind of appealing right now, considering all the alternatives
05:11   pitti   mdz: vs. managing the po files (and other stuff) aside from debs
05:11   mdz     it allows us to safely update translations without touching the package itself
05:11   sabdfl  F2-3:  we should expect upwards of 200-500 languages within a year
05:11   pitti   I still think that F4 is the sanest variant
05:11   smurfix_        pitti: that kills every otion but F4 (and F6 ;-)
05:11   pitti   with some ideas from F3-2
05:12   pitti   i. e. download new translations to e. g. /var/cache/locale
05:12   mdz     pitti: did you make some estimates for disk space requirements?
05:12   sabdfl  Kamion: you mean, taking /usr/share/local/ out of dpkg control, and managing it separately?
05:12   pitti   and provide this as an alternative gettext root
05:12   Kamion  sabdfl: with respect I find that hard to believe, considering 10+ years of free software translation work that have produced maybe 20 languages with plausible coverage
05:12   mdz     200-500 is quite a lot more than any package we currently ship
05:12   pitti   mdz: the user would only download the languages and translations that he wants
05:12   sabdfl  Kamion, mdz: these are clear goals of ubuntu/rosetta
05:13   pitti   mdz: so disk space is minimal, without any deb overhead
05:13   Kamion  sabdfl: I know, I'm just saying I find it hard to believe.
05:13   pitti   mdz: and overhead of unwanted translations
05:13   Mithrandir      pitti: why would F2 include conflicts?  Replaces should be enough.
05:13   mdz     sabdfl: yes, I'm saying that we need data to know what that would really look like in terms of resources
05:13   sabdfl  fair nuff, just so you understand that's why i'm pushing this so hard now
05:13   pitti   Mithrandir: I think it's a bad idea to have a language pack replace a complete application deb
05:14   Mithrandir      pitti: Replaces: doesn't mean that.
05:14   pitti   sabdfl: I do _not_ propose to change anything in /usr/share/locale, btw
05:14   sabdfl  what about this?
05:14   Kamion  sabdfl: don't think /usr/share/locale/ should be taken out of dpkg's control; pitti's F4 suggests somewhere in /var
05:14   pitti   sabdfl: I only want to have additional translatiosn in /var/cache/locale
05:14   sabdfl  could we update the ubuntu gettext to look in *two* locations? and take the newer?
05:14   pitti   Kamion: yes
05:14   sabdfl  language packas would go in /usr/share/langpack/
05:15   pitti   sabdfl: this should be easy for the majority of packages
05:15   mdz     gnome-panel has 70 locales totalling 4.2M
05:15   pitti   sabdfl: however, we need to manually fix packages which link gettext statically
Mithrandir thinks /usr/local/share/locale would be the right place, but that's a detail.
05:15   Kamion  avoiding any location that is currently used produces the huge benefit that we do not become instantly incompatible with everyone
05:15   sabdfl  then the local build could go into a diff location
05:15   Kamion  Mithrandir: it's still distro-managed though
05:15   smurfix_        pitti: that makes sense anyway
05:16   pitti   Mithrandir: I would favor /var; /usr might be r/o
05:16   sabdfl  the main thing being that the langpack would not put files anywhere a locally built deb would expect to be able to do
05:16   pitti   My idea is to have a language pack which controls the downloading and installation
05:16   Mithrandir      pitti: depends on where the langpacks comes from -- if they are .debs, then /usr is r/o => you lose.
05:16   mdz     I like pitti's idea of treating the two pools of translations separately
05:16   pitti   Mithrandir: as I said, I don't think it's a good idea to put updated translations in deb
05:17   mdz     but it highlights a question about the overall process
05:17   Mithrandir      pitti: why not?
05:17   mdz     is our goal that these translations be passed upstream?
05:17   sabdfl  mdz: yes, through rosetta
05:17   pitti   Mithrandir: I explained in detail in my list
05:17   Mithrandir      pitti: how else are you going to make sure that cruft is removed?
05:17   mdz     if so, upstream inherits a lot of these problems
05:17   pitti   Mithrandir: it's nothing but trouble
05:17   pitti   Mithrandir: this must be handled by the download manager (i. e. the language pack)
05:17   mdz     their source tarball grows, their default install size grows for source-based installs and for other packagers as well
05:17   Kamion  mdz: upstream don't really, Debian does though
05:17   smurfix_        mithrandir: either you have LOTS of debs or your update is a 99% no-op
05:17   pitti   Mithrandir: it shouldn't be too hard to sort out cruft in gettext files
05:18   Kamion  most upstreams don't seem to care about source tarball size :)
05:18   smurfix_        mithrandir: both is not nice
05:18   sabdfl  btw: bazaar 1.0 release in progress now
05:18   Mithrandir      pitti: I'm talking about cruft as in "handling languages which are no longer on the system".  You would be reinventing dpkg.
05:18   Kamion  how about /usr/share/locale/extra/ or something?
05:18   Mithrandir      Kamion: do people care about installed-size?
05:18   pitti   Mithrandir: maybe, but using dpkg is even worse...
05:19   Kamion  would be nice to keep within FHS directories
05:19   pitti   Mithrandir: deleting a language pack can just remove /var/cache/locale/<lang>
05:19   smurfix_        mithrandir: it's a somewhat different problem space
05:19   pitti   Mithrandir: s/delete/purge/
05:19   Kamion  or at least get this standardised somehow, since it sounds as if everyone will run into death-by-language-bloat soon
05:19   mdz     Mithrandir: if installed-size were not an issue, I think language packs would be a non-issue as well
05:19   mdz     well, almost
05:20   pitti   mdz: there are also troubles with mirrors and pacakge rebuilds
05:20   mdz     there would still be the issue of .deb size
05:20   pitti   mdz: installed-size of language packs is the least evil point
05:20   Mithrandir      mdz: I'm asking whether it's really an issue or whether we just have a bunch of whiners who can be ignored.  (To be harsh)
05:20   mdz     Mithrandir: any whiners are hypothetical at this stage
05:20   Kamion  Mithrandir: CD size is pretty hard and fast
05:20   mdz     we're extrapolating what would happen if the number of translations exploded
05:20   smurfix_        kamion: teaching dpkg to basically ignore some directories should be reasonably simple
05:21   mdz     smurfix: that would only address installed-size, and not .deb size
05:21   Kamion  smurfix_: that approach is pretty fragile and hard to reconfigure later though
05:21   Kamion  and, as mdz said
05:21   pitti   smurfix: why you want to do that?
05:21   sabdfl  easy, Mithrandir
05:21   Kamion  smurfix_: if somebody says "please can I have French translations now", you have to reinstall everything
05:21   Mithrandir      sabdfl: I didn't intend to stomp on any toes -- I was wondering if we were discussing a real problem or not. :)
05:21   mdz     we are
05:22   mdz     though we are trying to discuss it before we actually have it
05:22   pitti   Can we at least agree to drop the F2 options?
05:22   sabdfl  in terms of where i would like ubuntu to be in two years time, which is much more widely translated than rhat, suse or debian, yes this will be a real issue
05:22   pitti   i. e. extract translations during build into separate debs?
05:22   smurfix_        kamion: assuming that the files can't be downloaded from rosetta-or-whatever
05:22   sabdfl  fine by me
05:23   mdz     wait
05:23   mdz     F2 are the only options which address the problem of .deb size
05:23   mdz     oh, also F5
05:23   sabdfl  our build process will have to be different
05:23   pitti   mdz: ?
05:23   Mithrandir      mdz: F5 means the shipped debs are different from what you get from apt-get -b source $package, though.
05:24   sabdfl  in that we will not include the translations in a default .deb build
05:24   mdz     Mithrandir: yes
05:24   mdz     F5 has big problems in terms of package management
05:24   mdz     but it does address the problem of .deb size
05:24   Mithrandir      pitti: F3, F4 doesn't shave any size off the shipped debs.
05:24   pitti   mdz: the point is that _anything_ using debs created by us makes troble
05:24   elmo    F4 could be combined with a stripping
05:24   Mithrandir      pitti: which we want to do, in order not to hit the 650MB limit.
05:24   mdz     we could do a slight modification of F5 which results in having the same .debs in the archive and on the CD
05:24   elmo    F4+strip down to English +whatever we can fit is by far the most sane option IMHO
05:25   pitti   Mithrandir: yes, but _we_ have to create debs regularly
05:25   mdz     local builds break
05:25   pitti   Mithrandir: as opposed to F4, when the _user_ decices when to download what
05:25   Kamion  sabdfl: I really think patching the source packages is the only correct way to modify the build process
05:25   sabdfl  Kamion: yes, i expect that
05:25   pitti   Mithrandir: this simplifies both building and downloading a lot
05:25   Mithrandir      Kamion: I agree with you on that.
05:25   Kamion  sabdfl: ok, good, alleviates my major concern :)
05:25   mdz     sabdfl: based on our experience so far with merges, that would be a prohibitively large workload for our team
05:26   elmo    mdz: okay, F4 +strip in binary and source
05:26   pitti   Kamion: but I think if we can manage without patching source packages at all, this would be even better
05:26   mdz     patching every source package is not an option for us
05:26   smurfix_        elmo: why both binary and source? Agreeing on either makes more sense imho
05:27   mdz     elmo: hmm
05:27   Kamion  mdz: the fact that a source package specifies what goes into its binary packages is an invariant; breaking that just for scheduling reasons is so wrong
05:27   elmo    mdz: it's not every source, nly what's on the CD
05:27   elmo    we can't not patch what's on the CD if you want local builds to work, AFAICS
05:27   Kamion  if we're going to modify the build process then that change must be in Ubuntu
05:27   Kamion  (i.e. not buildds)
05:27   mdz     elmo: I think I like that very much
05:27   elmo    smurfix: eh?  the binary have to be modified to get the space down, the source has to be modified, if you want a local rebuild to procue an installable .deb
05:27   smurfix_        I'd prefer mangling the binaries when they arrive from the autobuilder.
05:27   mdz     (F4 + stripping)
05:28   Kamion  smurfix_: so wrong, so wrong
05:28   mdz     that avoids the conflict issues with local builds
05:28   pitti   Am I right that stripping and F4 are actually completely orthogonal?
05:28   Kamion  think I agree with elmo here
05:29   pitti   so we can discuss that independently?
05:29   Mithrandir      mdz: and then leaving /var/cache/locales non-dpkg-managed?
05:29   mdz     pitti: yes
05:29   mdz     good plan
05:29   mdz     we can decide separately how to best remove translations from .debs, and how to provide the translations separately
05:29   smurfix_        elmo: why would a local rebuild create a conflict?
05:29   mdz     the former is the difficult bit
05:30   Keybuk  stripping how?
05:30   mvo_    I think the later is tricky too :)
05:30   Kamion  mdz: adding dh_striplocales (say) to debian/rules wouldn't be that bad, would it?
05:30   Kamion  if it's a one-liner ...
05:30   Keybuk  at buildd-time, or in the source package?
05:30   elmo    smurfix: oh, okay, it wouldn't necessarily, but it wouldn create a different .deb, and there's a lot to be said for idempotency of builds
05:30   Mithrandir      Kamion: aka rm -rf debian/$packagename/usr/share/locale ? :)
05:30   sabdfl  it should be a one-time one-liner
05:30   pitti   mdz: stripping binary debs is easy, but stipping source packages? We have to change every orig.tar.gz...
05:31   Kamion  pitti: and then various upstream projects will accuse us of forking
05:31   pitti   Kamion: how would that help to reduce the source package size?
05:31   Kamion  (c.f. openssh)
05:31   Keybuk  Kamion: the bit where it would go doesn't tend to change hugely -- though it'd trip on a major change to debian/rules and need manually fixing
05:31   Kamion  Keybuk: yeah, but it should be mostly automergeable
05:31   Keybuk  9/10 times
05:31   Keybuk  maybe 99/100 too
05:31   sabdfl  pitti: source package size is not the issue
05:32   pitti   elmo: do you actuallly mean "strip the source package" or "strip during building the debs from the source package"?
05:32   Kamion  sabdfl: source CDs are a reality, various people want them
05:32   elmo    pitti: latter
05:32   pitti   sabdfl: ah, okay. I misunderstood
05:32   pitti   elmo: okay, that's easier
05:32   mdz     Keybuk: is it feasible to detect the case where we _only_ made this change, and fast-track it?
05:32   elmo    kamion: point them at > 650Mb ISOs, what do we care?
05:32   sabdfl  Kamion: they can be multi-cd sets
05:32   Keybuk  mdz: yah
05:32   sabdfl  dvd
05:32   Kamion  sabdfl: they're already 3 CDs
05:32   Kamion  sabdfl: turning them into 30 seems a bit non-trivial
05:32   sabdfl  true :-)
05:32   Kamion  (3 CDs for hoary as it stands)
05:32   elmo    Kamion: don't you like a challenge??
05:33   Kamion  elmo: depends how fast you want auckland's disk space used up
05:33   mdz     last I heard, we weren't hurting for disk space
05:33   Keybuk  mdz: wish we were based on darcs rather than arch now though :p  could have a "add dh_striplocales before dh_builddeb" change token :p
05:33   sabdfl  ok, girls, focus ;-)
05:33   Kamion  mdz: we will be if daily CD builds multiply by ten in size
05:33   mdz     Kamion: I don't think we quite need _daily_ source CDs
05:34   Kamion  mdz: we already have them, 'cos building CDs daily is about the only sane way to make sure they keep working
05:34   pitti   before we discuss the details, is there anybody who thinks that strip+separate download is _not_ the solution?
05:34   sabdfl  i don't think we need to keep them around though
05:34   mdz     Kamion: if it's only as a test, we can throw away the result if the space become sa problem
05:34   mdz     what sabdfl  said
05:34   Kamion  mdz: true
05:34   Mithrandir      pitti: I wish they are handled by dpkg, but it seems people disagree with that, so.
05:35   mdz     so, considering that we must remove translations from .debs in order to maintain the One True CD, I think we only have two options for the "remove translations from .debs" piece
05:35   mdz     (source or binary)
05:35   pitti   Mithrandir: if you find a way to sanely handle translations in deb, tell it to us :-)
05:35   sabdfl  +1 source
05:35   mdz     source is more correct in every way from a pure technical perspective
05:35   pitti   Mithrandir: I would prefer proper debs from a cosmetic viewpoint too
05:36   pitti   mdz: binary is easier, but source cleaner
05:36   Kamion  source => change source package, binary => adjust binary package after build?
05:36   mdz     Kamion: correct
05:36   Mithrandir      pitti: they are just a bunch of files.  Wrap them up in a deb; make a daily deb of all the files for a language if you so desire.
05:36   Keybuk  source++
05:36   Kamion  definitely +1 source inasmuch as I don't get a TB vote
05:36   Keybuk  Mithrandir: "Dear Mirrors, hello, here's SOME DEB"
05:36   pitti   Mithrandir: either the mirrors or the number of debs would explode
05:36   mdz     however, source thrusts us into a situation of having modified several times as many packages as we currently do
05:37   smurfix_        hmm, if source, how do the stripped-off translations end up where people can download them?
05:37   mdz     so it is dependent on a greater level of merge automation
05:37   Mithrandir      Keybuk: one per language; the biggest language on my system is french, that is about 8MB, compresses down to about 2.7MB.
05:37   Keybuk  mdz: from 233 to "all of them" ?
05:37   pitti   mdz: we can modify debhelper to do it automatically
05:37   pitti   smurfix: but them into rosetta, I think
05:37   mdz     some have claimed that we would only modify the packages on the CD
05:37   Keybuk  Mithrandir: 200 languages, every day, half a gigabyte to each mirror each day
05:37   mdz     which I think would be about 3x as many packages as we currently modify
05:38   Mithrandir      Keybuk: only build if they have changed that day, then.
05:38   Kamion  I think modifying dh_builddeb or something around that level in Ubuntu might be just about acceptable
05:38   Mithrandir      Keybuk: I doubt we'll have changes to each language each day.
05:38   Kamion  although it's a bit worrying
05:38   Keybuk  Mithrandir: we upload source every day, leading to a language-pack change every day, for every language
05:38   mdz     Kamion: except for non-debhelper packages
05:38   mdz     Kamion: do you feel the same way about modifying dpkg-deb? ;-)
05:38   Kamion  mdz: yeah, pretty trivial number though
05:38   Kamion  mdz: I feel deep abhorrence for that idea; layering violation
05:38   Mithrandir      Keybuk: I thought translations were to be handled completely in rosetta?
05:39   mdz     Kamion: I agree, fwiw
05:39   pitti   Mithrandir: but you will have a daily change in at least one package, which would trigger a rebuild of the whole langpack
05:39   Kamion  debhelper is allowed to know about details of Debian policy as regards package layout; dpkg-deb is just an archiver
05:39   Mithrandir      pitti: uhm, why would that be?
05:39   Kamion  so s/Debian/Ubuntu/ and it looks barely tolerable
05:39   Kamion  the problem is that that would break rebuilds of Debian packages
05:39   Kamion  which would be a bad thing
05:39   elmo    WRT debhelper, why don't we just declare it build-essential for us, then we can dh_strip_locale to any package even if it doesn't use debhelper?
05:39   pitti   Mithrandir: if you have one deb per language, it just is like this
05:39   mdz     Kamion: how so?
05:39   mdz     I think all source-based solutions share that problem
05:40   pitti   Mithrandir: if you have one deb per language per package, the number of debs explodes
05:40   Kamion  mdz: no they don't, modifying our source packages doesn't
05:40   Kamion  i.e. our source packages say what we want them to do
05:40   mdz     Kamion: the source package modification doesn't make it into Debian
05:40   mdz     you said rebuilds of Debian packages
05:40   pitti   mdz: I wouldn't change dpkg-deb, only debhelper
05:40   Kamion  mdz: uh-huh, but people rebuild Debian packages on Ubuntu
05:40   Mithrandir      pitti: I'm talking about one deb per language.  And if we do get 200 languages, well, then we have half a gig extra data to push.
05:40   Kamion  mdz: e.g. backports
05:41   elmo    Mithrandir: dude, please read the logs for the last meeting, we went over this TO DEATH
05:41   Kamion  if some maintainer script in a Debian package does stuff to /usr/share/locale (and I believe some of them do), there'd be hell to pay
05:41   Keybuk  yeah, I think some people would panic if I uploaded dpkg with all the translations stripped out
05:41   pitti   Mithrandir: you have to push half a gig for _every_ language every day
05:41   Mithrandir      elmo: ok
05:41   Keybuk  I vote for dh_striplocales
05:41   Keybuk  or dh_stripmessages
05:41   pitti   Mithrandir: right now my /usr/share/locale is about 200 MB, but that will increase
05:41   Mithrandir      pitti: /query?
05:41   Kamion  Keybuk++
05:41   mdz     ok
05:42   pitti   Mithrandir: ?
05:42   mdz     so the broad solution of "modify the source package" seems to be favoured
05:42   smurfix_        kamion:not if we modify gettext and put the rosetta translations somewhere else
05:42   mdz     within that solution, we have some choices
05:42   mdz     - modify debian/rules
05:42   mdz     - modify a debhelper tool (and, if debhelper is not used by the package, debian/rules also)
05:42   mdz     - others?
05:42   pitti   mdz: + modify rules
05:43   Kamion  - add a debhelper tool to simplify the modification of debian/rules
05:43   fabbione        - something that is absolutely shared by every package?
05:43   pitti   mdz: least surprise
05:43   mdz     Kamion: let's consider the debian/rules option under the assumption that we'll make the change trivial
05:43   mdz     by whatever means
05:44   mdz     fabbione: the only thing that falls into that category is dpkg-deb, and that's been declared evil
05:44   fabbione        mdz: aren't we? ;)
05:44   mdz     modifying debian/rules is going to mean adding a build-dependency as well
05:45   mdz     in order to guarantee a consistent result
05:45   Keybuk  mdz: or modifying build-essential
Kamion  could live without the build-dep for the purposes of expediency ...
05:45   mdz     hmm, even to guarantee that it builds
05:45   mdz     modifying debhelper lets us forego the build-dep
05:45   Kamion  upload debhelper, wait for elmo/lamont to install it in all chroots, upload everything else
05:45   mdz     if it's built with a debhelper that doesn't contain the change, they just don't get a stripped package
05:45   elmo    modify build-essential ..
05:46   Kamion  if we add a new debhelper program then the build will fail if it's missing
05:46   Kamion  which is good enough
05:46   mdz     hmmm
05:46   mdz     and then the user is left scratching their head over why the package doesn't build
05:46   fabbione        lamont will kill somebody just the amount of mails he will get :-)
05:46   elmo    we need to ensure that if a user does 'apt-get build-dep gnupg', and tries to build ubuntu's source, it'll work
05:46   Kamion  mdz: hoary'll ship with this modified debhelper presumably
05:46   elmo    the only way to guarantee that is to modify our build-essential
05:47   mdz     we'd break building Ubuntu debs on Debian, unless our debhelper change goes upstream
05:47   Kamion  we could even ask joeyh to include the tool in mainstream debhelper
05:47   elmo    we could (haha) || true it, or enclose it in a if [ -e ... ]  type test
05:47   mdz     I think he'd accept the dh_striplocales variety
05:47   Kamion  mdz: we've already broken building a lot of Ubuntu .debs on Debian mind you
05:48   pitti   Kamion: essentially it would just throw away any po files, right?
05:48   mdz     perhaps not the dh_builddeb variety
05:48   Kamion  mdz: plenty of them require Ubuntu-specific build-deps
05:48   mdz     Kamion: really?
05:48   Kamion  I think dh_striplocales is more correct anyway
05:48   Kamion  mdz: sure, take GNOME
05:48   mdz     I thought there were quite few of those
05:48   Kamion  mdz: or our d-i
05:48   mdz     d-i, sure
05:49   mdz     Keybuk: didn't you have some numbers on how many packages don't use debhelper in warty/main?
05:49   Kamion  might be worth ignoring all packages without translations
05:49   haggai  I think you should try to find a solution that is upstreamable so that co-operative Debian maintainers can add the tool call to their source to reduce the debian->ubuntu diff
05:49   mdz     745/1022 I it looks like
05:49   mdz     Kamion: I think the plan is to translate those, too :-)
05:49   Keybuk  mdz: yeah, it came down about 30 don't
05:50   sabdfl  we have a hard limit at this stage on those which don't use gettext at all
05:50   Kamion  mdz: how about the ones without strings? :)
05:50   mdz     745/1022 have a build-dep on debhelper
05:50   elmo    wow, that's less than I expected
05:50   Keybuk  mdz: build-dep-indep too
05:50   sabdfl  ok, i think we have an emergin consensus
05:50   mdz     grep-dctrl -FBuild-Depends,Build-Depends-Indep -nsPackage debhelper
05:50   mdz     745
05:51   Keybuk  mdz: grep-dctrl in hoary is broken
05:51   mdz     oh, sweet
05:51   Keybuk  grep-dctrl -s Package -FBuild-Depends,Build-Depends-Indep
05:51   Keybuk  debhelper /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_hoary_main_source_Sources
05:51   Keybuk  | wc -l
05:51   Keybuk  952
05:51   Keybuk  Actually it catches over 90%, and of the remaining 98 packages only 28
05:51   Keybuk  contain translations anyway (using the "does it depend on gettext or
05:51   Keybuk  po-debconf" test).
05:51   pitti   sabdfl: I think so too, since the guys are already talking about specific details
05:51   Keybuk  --
05:51   pitti   sabdfl: but I think the general strategy works and is the sanest one
05:53   mdz     given some magic which saves us from the crushing merge workload, the preferred solution for stripping locales is a dh_striplocales one
05:54   pitti   agreed
05:54   sabdfl  ok, next!
05:54   mdz     can we really take that for granted at this stage?
05:55   mdz     language packs are a hoary goal
05:55   mdz     this level of divergence almost requires that we can put most of these merges on full automatic mode
05:55   mdz     including the upload
05:55   sabdfl  mdz: let's evaluate that after the first week of Mataro Sessions
05:56   sabdfl  we'll have a handle on hct at that point
05:56   Keybuk  hct doesn't help any more than mom
05:56   Keybuk  a little less, in fact, because it requires tla/baz to play nice
05:56   elmo    it's worth noting, that we're not running out of space yet either, so they may be a goal but they're not a 100% we can't ship without this
05:56   sabdfl  yes
05:56   mdz     elmo: however, it is a high priority item on Mark's hoary list
05:57   Keybuk  mom works by breaking apart, mutating and driving patch itself
05:57   mdz     which is fairly close to the same semantics
05:57   Keybuk  we don't get that power with arch
05:57   Kamion  (we have 55MB free on the powerpc CD, which is tightest)
05:57   mdz     Kamion: and we need to get a ppc64 kernel image onto it :-/
05:57   mdz     Keybuk: yet :-P
05:57   Kamion  mdz: should be OK ...
05:58   mdz     that'll eat about 15M
05:58   mdz     guestimating from the powerpc one
05:58   Kamion  mdz: depends which of powerpc/power3/power4 we need
05:58   Kamion  (can we bounty benh to fix the MMU crap that means we need those three? :-/)
05:59   elmo    can't we pay/beg/bribe/hold hostage someone to get rid of the stupid sub-arch split... doh
05:59   mdz     ok, so regarding stripping locales, we're fairly settled on dh_striplocales, on the assumption that Keybuk can provide the level of support we need for the merges
05:59   pitti   sabdfl: is there actually a "next" item on the list?
05:59   mdz     Kamion: let's talk about that a bit later
06:00   mdz     Kamion: doesn't sound unreasonable at first hearing
06:00   mdz     pitti: yes
06:00   mdz     the second half is the approach for providing the translations in separate .debs
06:00   mdz     it sounds like we were moving in the following direction there:
06:01   mdz     - buildds collect the translations via dh_striplocales and drop them into a pool
06:01   mdz     - some automated process runs periodically and builds .debs from the pool
06:01   elmo    debs are crack for this
06:01   mdz     - the .debs contain translations in some directory which is not /usr/share/locale
06:01   mdz     oh?  was that decided at the last meeting?
06:01   elmo    I dunno, I'm just saying what I think
06:02   mdz     ok, s/\.debs/language packs/ for now
06:02   pitti   mdz: My idea is to have language-support-XX manage the download, install, and housekeeping
06:02   pitti   mdz: but not stuffing the po files into a deb itself
06:02   mvo_    I think the problems reamins the same whatever format we use
06:02   pitti   a reasonable compromise would be to build a deb on the fly on the user's machine
06:02   mdz     pitti: I think that introduces much more complexity than we want
06:03   pitti   mdz: but if we push debs, we have all these problems
06:03   mvo_    if we don't use deb we need to have package manager like functionality in a language-pack-get tool
06:03   pitti   mdz: assembling debs on the fly on user's hosts seems much more sane then
06:03   Mithrandir      pitti: I'm not too comfortable with that -- l-s-xx needs to be able to acquire, unpack, record what's installed and uninstall the files later.
06:03   mdz     pitti: I think that has the same problems
06:03   pitti   mvo_: but only downloading, copying and cleaning
06:03   mdz     there is a lot of value in having a language pack blob which actually contains the translations
06:03   pitti   Mithrandir: why record?
06:03   mdz     which can be placed on the CD, passed around by the user, etc.
06:04   Mithrandir      pitti: it needs to know what it has installed so it knows what it should update.
06:04   pitti   mdz: but the whole point of all this is to avoid debs in the first place
06:04   pitti   at least debs we provide on our mirrors
06:04   Mithrandir      elmo: you think .deb files are insane because of the mirror load?
06:04   mdz     it is?
06:04   Mithrandir      elmo: or anything else too?
06:05   elmo    Mithrandir: Packages files are useless for them and too large
06:05   elmo    and yes, I don't desperately want them in the pool proper
06:05   pitti   Mithrandir: not only because of the mirror load, also because of the download redundancy
06:05   pitti   The user should be able to download only the translations for packages he actually uses
06:05   mdz     elmo: don't desperately want, or desperately don't want?
06:05   haggai  could new functionality for registering conffiles etc with dpkg (Keybuk?) be used to register new translations with the dpkg database?
06:06   mdz     pitti: what it sounds like you're describing is a .deb which can't be installed without access to <someplace>.ubuntu.com
06:06   pitti   mdz: ?
06:06   mvo_    when will he download them? will he need to do a apt-get upgrade and a lang-pack upgrade?
06:06   mdz     pitti: <pitti> mdz: My idea is to have language-support-XX manage the download, install, and housekeeping
06:06   pitti   mdz: l-s-XX would download the translations, create a deb and isntall it
06:06   mdz     pitti: correct
06:06   Mithrandir      mvo_: think install from cd, for instance.
06:06   pitti   mdz: you need access to Rosetta, right
06:06   elmo    mdz: depends what format the debs are, one per source or one per lang?
06:06   mdz     that means that installing l-s-XX would require that the target system be able to talk to <someplace>.ubuntu.com
06:07   elmo    mdz: the latter can  be over my dead or P45-ed body
06:07   pitti   mdz: but you need that anyway to get _any_ update
06:07   mdz     which, in a wide variety of environments, just won't be true
06:07   Keybuk  haggai: doesn't work, conffiles still need a replaces:/--force-overwrite
06:07   pitti   mdz: building the deb can happen on the client
06:07   smurfix_        mithrandir: so the files are on CD instead of being downloaded
06:07   elmo    mdz: the former.. I'm pretty unhappy about too, but I'd need to look at proper stats to be sure
06:07   mdz     pitti: that isn't true, users have lots of ways of moving around .deb files
06:07   mdz     pitti: the user needs to be able to insert a CD and obtain a language pack that way
06:07   pitti   mdz: they can build the deb on a place with net access
06:08   haggai  Keybuk: I didn't mean that.  I meant, the ability to register a new file with the language-support deb
06:08   pitti   mdz: they can as well just copy the po tree to their computer (with the help of l-s-xx)
06:08   mdz     pitti: this defeats the purpose of providing a plug-and-play language pack
06:09   mdz     what is the problem that we would hope to solve with such a solution?
06:09   pitti   mdz: I still don't understand the problem
06:09   mdz     large downloads for people tracking the development branch?
06:09   pitti   mdz: where the deb is built, or who does it makes no difference
06:09   mdz     pitti: the problem is that it's essentially an installer .deb, and installer .debs are inherently problematic
06:09   pitti   mdz: we can put it onto a CD as well
06:09   sabdfl  we can accept a certain lag during the freeze
06:09   Mithrandir      mdz: large downloads for people who are running stable but want updated translations, I think.
06:09   sabdfl  so, say, langpacks are updated weekly rather than daily
06:10   mdz     sabdfl: right
06:10   mvo_    sabdfl: sounds good too me
06:10   sabdfl  i think elmo is right that there are benefits to avoiding the pool altogether for this
06:10   mdz     and we can provide a separate repository with daily updated language packs for those who really want them
06:10   pitti   guys, it is just insane to publish translation debs in regular intervals
06:10   smurfix_        why do we need .debs anyway?
06:10   sabdfl  as is mithrandir in that it's a small step towards reinventing dpkg :-)
06:10   mdz     if the problem is that the .deb is too large, we break it up
06:10   Mithrandir      .. you know, dpkg started out as a perl script.  We could reinvent it in python.
06:10   sabdfl  but... fundamentally, a langmgr on an ubuntu desktop needs to keep track of the languages that desktop needs, and sync in the data
06:10   pitti   let the user choose which translations he want, at which intervals and give him an easy way to update his system with a "pull" strategy
06:10   sabdfl  it might even use rsync!
06:11   Kamion  Mithrandir: actually it was a shell script before that ...
06:11   Kamion  (IIRC)
06:11   mdz     sabdfl: why does it need to do that?
06:11   Keybuk  you could always use the deb + delta idea ...
06:11   pitti   mdz: what do you mean by installer deb?
06:11   mdz     sabdfl: the user selects their languages, and from that point forward it's a simple upgrade
06:11   mvo_    Keybuk: +1
06:12   mdz     pitti: a .deb which doesn't contain the data that it is meant to provide, but instead downloads it over the network (or requires that the user do so)
06:12   pitti   mdz: with per-package debs or per-language?
06:12   mdz     Keybuk: not for hoary we can't
06:12   sabdfl  mdz: you mean, if we have an ubuntu-xx langpack in the pool? elmo hates that idea, and he's bigger than me, and i happen to agree with him
06:12   pitti   mdz: you can also install readymade translation debs
06:12   mdz     sabdfl: I'm going to go out on a limb and ask him to justify his hatred :-P
06:12   mdz     sabdfl: you, too, if you agree
06:12   pitti   mdz: it should be _possible_ to assemble them on the fly, not required
06:12   sabdfl  the langpack debs will drive the mirrors crazy
06:13   sabdfl  we can publish the translations as an rsyncable tree
06:13   Mithrandir      sabdfl: if they're only updated when they need to, or weekly or something as well?
06:13   mdz     as I said
06:13   mdz     if the problem is that the .debs are too big
06:13   mdz     then we break them up
06:13   elmo    mdz: it's utter crack.  if you change one translation in one source package, this n hundred Mb unrsyncable (and not normally rsynce danyway) .deb changes
06:13   sabdfl  then dpkg/apt need to scale to more debs than they should have to
06:13   Mithrandir      mdz: then you get a huge amount of them.
06:13   Kamion  mdz: doesn't help, lots of them will generally change at once
06:13   pitti   ARGH
06:13   mdz     depends on how you break them up
06:13   pitti   this is the very same discussion we had last time
06:14   Kamion  mdz: either you have a source message which changes and lots of translations change, or you have a common translation of a frequently occurring string that changes
06:14   Kamion  mdz: you can't win either way
06:14   elmo    pitti: yes, I know. let's petition sabdfl to not let mdz go on holiday and miss TB meetings ;-)
06:14   pitti   pushing debs to mirrors is crack
06:14   sabdfl  elmo: only if you promise not to go on holiday too ;-)
06:14   smurfix_        I agree with sabdfl. Just use rsync and ignore all files belonging to packages you don't have installed. Problem solved.
06:15   pitti   smurfix: when using /var/cache/locale, we don't even have the possibility of hitting pacakge conflicts
06:15   sabdfl  smurfix_ it's easier than that, since the desktop install generally installs a solid well-defined set of packages
06:15   sabdfl  we only do this for those
06:15   mdz     I think you guys are optimising for the development branch case
06:15   mdz     when we need to optimise for the stable release case
06:15   sabdfl  yes, understood
06:15   mdz     these problems are non-issues for the stable release
06:15   pitti   mdz: you cannot force users to download huge amounts of debs for small translation changes
06:16   Mithrandir      sabdfl: you'd need to ship something to rsync off on the cd, then.  That is a lot more work than shipping one more deb.
06:16   pitti   mdz: s/cannot/it wouldn't be the best solution/
06:16   sabdfl  but i /really/ dislike the idea of switching something this big on only at release time, we discussed that last time
06:16   elmo    mdz: dude, this is not "optimization", this is "having a mirror network vs. not"
06:16   smurfix_        pitti: exactly. local packages use /usr/share. So check there too, and pick the newer one.
06:16   elmo    mdz: if we implemented this with even the number of languages/translations we have today, we'd have zero mirrors within a month
06:16   elmo    hell, _I_ wouldn't mirror us over the GB LAN :-P
06:17   sabdfl  elmo: i love how you tell the truth :-p
06:17   smurfix_        mithrandir: so park the rsync-master file tree onto the cd ..?
06:17   mdz     elmo: ok, I understand.  let's try to change our approach to adapt to that, rather than throwing it away
06:17   Mithrandir      smurfix_: have you touched debian-cd?
06:17   Kamion  smurfix_: er ... uh-huh.
06:17   mdz     I'll say again that this is strictly a development branch issue
06:17   Mithrandir      smurfix_: doing that would be very icky and interesting.
06:17   Mithrandir      smurfix_: I do not think Kamion would like you afterwards. :P
06:17   pitti   mdz: so we need a system that is both able to install premade language debs, prepare these and also update directly from the network
06:17   mdz     and I do not think that we should sacrifice having the feature look the way it should in the final release
06:18   Mithrandir      mdz: I'm worried that elmo is so much against it, though..
06:18   smurfix_        kamion: On the language-pack CD of course. Maybe packaged inside a cpio tree if too many small files are too icky.
06:19   Kamion  smurfix_: I think all this is highly premature
06:19   Mithrandir      smurfix_: that sounds like you're reinventing rpm rather than dpkg, then.
06:19   mdz     Mithrandir: the issues that elmo raises apply only to the development branch, and I think the use case is different enough that we may need to solve the problem in two slightly different ways
06:19   Mithrandir      mdz: mirrors mirror all, though
06:20   Kamion  smurfix_: language pack CD structures aren't something we've even started to think about yet, and I'm not convinced that they're something we ought to be generating centrally anyway
06:20   pitti   mdz: why, if stable users should be able to download their daily translation updates as well, it affects stable too
06:20   mdz     we are not going to force a huge delta increase on mirrors
06:20   Kamion  better to give people an easy way to build an Ubuntu CD for their language
06:20   elmo    mdz: dude, it's not just the development branch.  what about a security update that changes a translation?
06:20   mdz     the sky is NOT falling
06:20   mdz     elmo: they don't
06:20   elmo    bullshit
06:20   elmo    you've uploaded new upstream versions before - it's rare, but it happens
06:20   Kamion  mdz: also security issues exist in translations; consider format string errors
06:21   mdz     if we have a clusterfuck like that, we put the changed translations into the security update .deb
06:21   smurfix_        Well, I'm brainstorming on "how could this work" rather than "how do we make .debs to make it work", so ...
06:21   elmo    mdz: and what Replaces: the langpack?
06:22   Kamion  and if the langpack is a .deb we then run into the non-commutativity of Replaces:
06:22   Kamion  (muttermutterdpkgbugsmuttermutter)
06:22   mdz     elmo: uses a different path
06:22   Mithrandir      versioned conflicts with the broken version of the langpack. (Yes, it's broken, but it works-ish)
06:22   elmo    mdz: and how do apps know which to use?
06:22   Kamion  Mithrandir: urgh
06:22   Kamion  Mithrandir: yay for removing langpacks while upgrading packages
06:23   mdz     elmo: they prefer the one provided by the package, if present (generally not)
06:23   Mithrandir      Kamion: doesn't apt/dpkg DTRT and upgrade?
06:23   smurfix_        ... or the one timestamped newer.
06:23   elmo    mdz: that technology exists for locales already?
06:23   mdz     elmo: if it doesn't, it seems completely trivial to implement
06:23   Kamion  Mithrandir: not if an updated langpack isn't available yet
06:24   Mithrandir      Kamion: then we have to make sure it's available.  (Yes, I know that's not trivial either)
06:24   Keybuk  Mithrandir: dpkg does not upgrade on << conflicts.
06:24   Mithrandir      Keybuk: what about apt and dselect?
06:25   mdz     Mithrandir: depends
06:25   elmo    mdz: *shrug* ok then.  so how/when do you propose to transition to the stable model as part of the development process?
06:26   mdz     depends on how we approach the development-branch model
06:26   elmo    I still think you're trying to fit a square into a round hole, 'cos the round hole is familiar and looks pretty, but that's probably just me
06:26   pitti   err, what is our "stable model"? and why it should be any different from the development one?
06:26   mdz     pitti: put yourself in end-user mode for a moment and consider the problem
06:26   mdz     "I want my Ubuntu to speak <language>"
06:26   pitti   mdz: stable versions do not have any fewer updates than unstable ones
06:26   elmo    oh, hang on
06:26   elmo    mdz this doesn't work
06:27   mdz     what's "this"?
06:27   elmo    mdz: one of the reasons for all this crack, is sabdfl's desire to have translation updates out of bound
06:27   elmo    i.e. in stable, still get translation updates
06:27   pitti   mdz: apt-get install l-s-<language> with a net connection works
06:27   elmo    so there is no stable model
06:27   pitti   mdz: if you don't have a net connection, make it easy to copy the files from another host which has
06:27   pitti   mdz: but that is orthogonal to the particular solution anyway
06:28   Mithrandir      elmo: push updated translation-debs to warty-updates once a week?
06:28   mdz     pitti: "you must have Internet access in order to do that, otherwise use this manual process"?
06:28   sabdfl  mdz: what elmo said. but not about the pretty familiar round hole
06:28   mdz     that answer won't fly for a lot of deployments
06:28   pitti   mdz: how do you want to download stuff without net?
06:28   pitti   mdz: it doesn't matter if you download debs or po file tarballs
06:28   pitti   mdz: "you" == the language pack which manages the stuff
06:29   mdz     sabdfl: what's the basis for updating stable with new translations, as opposed to doing the translations as part of the freeze process, as GNOME does?
06:29   elmo    Mithrandir: that has the "mirror admins lynch you" problem all over again
06:29   Mithrandir      elmo: yes, that might be a slight problem.
06:29   sabdfl  mdz: reality
06:29   mdz     pitti: there is a big difference between a packaging system which downloads packages, and a package which downloads .po files
06:29   Mithrandir      pitti: we end up having to reinvent dpkg to grab the translations off $MEDIA
06:29   pitti   mdz: we can include the current language snapshots on the CD too if we want
06:30   mdz     pitti: you open yourself to a lot of site-specific problems
06:30   mdz     outbound access policy, proxy authentication, ...
06:30   pitti   mdz: but we can use debs to hold the actual updates
06:30   mdz     pitti: and have the package copy them from the CD when it's installed?
06:31   mdz     then suddenly you need to implement this in apt
06:31   pitti   mdz: the point is not to avoid debs, but to avoid building and pushing them to our servers regularly
06:31   pitti   mdz: as I said, you can assemble the debs either on the fly or use premade ones
06:31   Mithrandir      would having a separate repository for translations be complete crack?
06:31   mdz     if it's a problem for them to be pushed to the mirrors regularly, then may I make the bold suggestion that we, well, don't push them to the mirrors regularly?
06:31   pitti   mdz: create them on the fly in rosetta?
06:32   pitti   mdz: when an user wants to update his translations?
06:32   elmo    mdz: or how about this for a bold suggestion: don't insist on sticking with a format that's so inflexible we can neither split them up, nor ship them in one big lump, and that way get to ship them as often as we want
06:32   mdz     pitti: what would the system look like which would decide whether to create a .deb or use an existing one?  which piece of the package management system would it inhabit?
06:33   mdz     what is the proposed alternative transport format?  a few million .po files?
06:33   smurfix_        mdz: million?
06:33   pitti   mdz: ^ that is actually the most flexible one
06:33   pitti   mdz: the user needs to download exactly the po files he needs, nothing else
06:33   elmo    per-source or per-binary package lumps of .po files is an alternative too
06:33   pitti   mdz: but still, having deb as transport format is not a bad idea on itself
06:34   mdz     smurfix: O(packages*languages)
06:34   mdz     which is O(thousands*hundreds) at least
06:34   pitti   mdz: the bad idea is not the deb format, but to include the debs into our normal archive structure
06:34   mdz     elmo: how exactly is a per-source or per-binary lump of .po files better or worse than per-source or per-binary .debs?
06:34   pitti   mdz: re your first question: the deb would be recreated if the user presses "update language"
06:35   pitti   mdz: of course with network access
06:35   elmo    mdz: by definition, they wouldn't be in the pool or bloating the Packages files
06:35   pitti   mdz: alternatively the user can install an updated deb from somewhere else
06:35   mdz     elmo: instead, they would be in a separate "pool" with its own index file and bloat that?
06:35   smurfix_        mdz: end users won't see that many files and mirrors can use more scalable distribution methods if necessary.
06:35   elmo    mdz: if you use .debs, yes :P
06:36   mdz     elmo: or anything else
06:36   pitti   mdz: if you download po files direcly, there is no need for index files at all
06:36   mdz     storing the .po files individually doesn't scale at all
06:36   pitti   mdz: why not?
06:36   mdz     aggregation is a must
06:36   mdz     pitti: how will you determine which ones to download?
06:36   pitti   mdz: timestamps?
06:37   mdz     pitti: you will do thousands and thousands of HTTP HEAD requests?
06:37   Keybuk  sorry guys, I'm going to have to bow out now
06:37   mdz     or download an index with every .po file in it?
06:37   pitti   mdz: not necessarily
06:37   Keybuk  have to pack for both London, Mataro and Christmas-week tonight
06:37   pitti   mdz: you can tell rosetta "give me all translations newer than date x"
06:37   pitti   mdz: rosetta would give you a tarball-like answer
06:38   pitti   mdz: the timestamps can be sent out by rosetta as well
06:38   pitti   mdz: so we don't need to rely on clocks
06:38   pitti   mdz: or peek on po files
06:38   mvo_    pitti: I'm not sure if this will not hit rosetta way too hard if a lot of users use our system
06:38   pitti   mdz: we just store the last timestamp/index mark/whatever of the last update
06:38   smurfix_        if http head doesn't scale, then don't use it ... sounds simple. :-P
06:38   pitti   mvo_: if users start to download translations daily, it doesn't matter if we hit rosetta or archive
06:39   Mithrandir      pitti: rosetta isn't mirrored.
06:39   mdz     smurfix: the point is that we don't have a choice in whether to aggregate .po files into larger blobs
06:39   mdz     which seems to be the idea that everyone is attacking
06:39   pitti   Mithrandir: then let's mirror it :-)
06:39   elmo    mdz: no, what I'm saying is that putting these in .debs in the archive is not a sane solution, given the requirements
06:39   pitti   mdz: but the aggregation would just be temporary as a download armor
06:40   pitti   mdz: s/armor/transport format/
06:40   Mithrandir      elmo: do you think putting them somewhere which is not mirrored would be as bad?
06:40   pitti   Mithrandir: if we don't have the bandwidth to support daily translation downloads, then there is no solution at all, I think
06:41   Mithrandir      if so, the language-pack tool could just be a small shell script which wraps apt similar to d-i's build process.
06:41   mdz     elmo: how do you propose that we provide them as a portable, CD-friendly, installable, removable chunk of data, then?
06:41   pitti   same transport format as for download, I think?
06:41   Mithrandir      pitti: putting plain files on CDs is painful, .debs are easy. :)
06:41   pitti   all translations later than 0
06:41   elmo    mdz: whatever you want dude, I'm not trying to impose a solution on you or pretend to be clever enough to have even close to a good one.  I'm just trying to tell you what is and isn't possible and/or sane from a archive/mirror perspective
06:42   pitti   Mithrandir: so wrap it up in a deb for my sake
06:42   mdz     elmo: the message I'm getting is that we need something which provides .deb-like semantics without sending mirrors running away screaming
06:43   mdz     even if we accept that we need a delta-update mechanism
06:43   pitti   mdz: maybe we should think a bit about this particular problem before continuing
06:43   mdz     we need an initial blob that we ship on CDs
06:43   mdz     pitti: aw, it hasn't even been two hours yet :-)
06:43   smurfix_        the transport blob on cd would just have a few files you don't want *shrug*
06:44   pitti   mdz: the cd can wrap up rosetta's transport format into a deb
06:44   mdz     how do we mirror translations, if not as .debs?
06:44   pitti   mdz: we should rather mirror rosetta's database
06:44   smurfix_        mdz: a replicated datavase?
06:44   Kamion  Mithrandir: .debs not in the main archive are no easier than random files, TBH
06:44   mdz     smurfix: say what?
06:44   smurfix_        database
06:45   Kamion  Mithrandir: at least not significantly
06:45   Mithrandir      Kamion: they are.
06:45   mdz     smurfix: I understood that, but what do you mean?
06:45   mdz     smurfix: "run a copy of this application"?
06:45   Mithrandir      Kamion: just duplicate the code to do security patches on the CD.
06:45   Kamion  Mithrandir: it's pretty trivial to get debian-cd to copy random files onto a CD
06:45   Kamion  Mithrandir: I know how to do it
06:45   Kamion  Mithrandir: the question is doing anything useful with them at the other end
06:45   mdz     anything which is not a flat file is not going to be mirrored to an extent even close to the current archive
06:46   mdz     let's explore that a bit
06:46   mdz     so we have some random data on the CD
06:46   mdz     well, not so random
06:46   mdz     translation data
06:46   mdz     which we need to install on the system
06:46   pitti   mdz: but with any aggregation you will loose the flexibility of individually updating po files
06:46   Kamion  mdz: I'm not necessarily saying it's a good idea, just that it's not impossible
06:47   elmo    mdz: it doesn't necessarily need to be - the hit on translations is going to be an order of magnitude less than archive for some time tocome, and if we come up with something that doesn't involve databases, we will get mirrors for it in time
06:47   Mithrandir      pitti: yes, you will.  Some overhead is fine.
06:47   pitti   mdz: the blob could be put into something similar to /var/cache/apt/archives, and l-s-XX could use that instead of downloading
06:47   mdz     elmo: something that doesn't involve databases == flat file, no/
06:47   mdz     ?
06:47   smurfix_        mdz: it's one possible solution. Whether it's actually feasible for mirrors to run mysql-or-whatever is a different question, which I'm not qualified to answer.
06:47   mdz     pitti: so d-i would do this copying?
06:47   elmo    mdz: no, I mean doesn't invovle replicating databases..
06:47   pitti   mdz: it probably has
06:48   pitti   mdz: similar to archive-copier
06:48   mdz     smurfix: I can say, even in near-complete ignorance of the existing mirrors, that it is not feasible
06:48   elmo    mdz: i.e. anything that can be mirrored via http, ftp and/or rsync
06:48   pitti   mdz: if l-s-XX sees that something is already in the cache, it doesn't need to contact rosetta
06:48   mdz     elmo: agreed
06:48   pitti   mdz: just an idea...
06:48   mdz     pitti: what about the case where I have installed the system, and then later want to install an additional language?
06:49   pitti   mdz: without downloading?
06:49   mdz     pitti: right, say I have an Ubuntu DVD which contains translations for many languages
06:49   Kamion  I would like to avoid piling more and more stuff into d-i which we don't have an interface for changing in the real system
06:50   pitti   mdz: same technology as with the initial install, I think, but we need an easy interface for that
06:50   Kamion  d-i should certainly take care of installing your primary language, but it's not going to solve the whole problem for you
06:50   pitti   Kamion: in fact d-i shouldn't bother about these blobs at all
06:50   pitti   Kamion: d-i should just install l-s-XX and be fine
06:50   Kamion  pitti: mkay
06:51   pitti   the lang packs should have the knowledge of getting their blobs
06:51   Kamion  pitti: but I think you'll find it needs to bother for net-less installs
06:51   pitti   Kamion: at least this seems to be more consistent
06:51   Kamion  pitti: at least at the archive-copier level or similar
06:51   pitti   right
06:51   mdz     pitti: the language pack will not be able to get data from the CD
06:51   pitti   not?
06:51   Kamion  pitti: the CD's ejected
06:51   mdz     I would object to reimplementing apt-cdrom in the language pack
06:51   pitti   mdz: me too
06:52   pitti   maybe archive-copier should be extended then
06:52   pitti   I don't have a very good solution without thinking about it for at least some hours
06:52   mdz     the reason that I'm hesitant to give up on .debs entirely is that so many of these problems simply go away for .debs
06:52   pitti   it's all ad-hoc solutions
06:52   pitti   mdz: then let's find a solution which uses debs as transport format
06:52   mdz     we even have gnome-app-install as a nice interface for add/remove
06:53   pitti   mdz: use debs as wrapper, but don't publish them onto our normal archive
06:53   mdz     I'm at a disadvantage, having missed the previous meeting
06:53   mdz     was a consensus reached there that .debs really aren't the way to go?
06:53   elmo    no
06:53   pitti   mdz: well, at the previous meeting we argued about extracting po files during build and shipping the debs in our archive
06:53   pitti   mdz: no
06:53   mdz     if we move away from .debs, we trade the delta-update problem for a bunch of other problems which aren't necessarily less
06:54   pitti   mdz: wouldn't that be cured by allowing debs to be built on the fly?
06:54   pitti   mdz: allowing, not requiring
06:54   pitti   mdz: initial translations would be on the CD as normal debs
06:54   mdz     pitti: I'm not clear on exactly what the debs-on-the-fly solution would look like, can you describe it?
06:55   pitti   but upgrades would assemble a new deb based on existing and downloaded translations
06:55   pitti   this new deb is then installed normally as a new revision of the old one
06:55   pitti   something along that line
06:55   mdz     what piece of software would be responsible for building the .debs?
06:55   pitti   the langpack, Isuppose
06:55   mdz     the language pack deb?
06:55   pitti   it would depend on dpkg-dev
06:56   pitti   it should be relatively easy to pack together a bunch of po files with a boilerplate control directory
06:56   mdz     if you're thinking of invoking dpkg --install in postinst or something like that, I think there are solid technical reasons not to do that
06:57   pitti   mdz: no, not necessarily in postinst
06:57   Kamion  can't be done in postinsts
06:57   mdz     definitely can't be done now, and good reasons not to fix it
06:57   pitti   mdz: of course we have to delay that somehow
06:57   mdz     pitti: dpkg post-hooks or some such?
06:57   pitti   mdz: I don't know instantly
06:57   pitti   mdz: but I feel that there surely is a sane solution for that
06:58   pitti   mdz: btw, creating the deb is not done in the postinst
06:58   pitti   mdz: it is done in sudo update-language-pack or so
06:58   pitti   mdz: and this could install the deb
06:58   mdz     pitti: it seems to me that you end up solving most of the .deb delta update problem in the process
06:58   mdz     in which case we should just solve the .deb delta update problem and be done with it
06:58   pitti   mdz: we just have to find a way to call update-lang-pack
06:59   pitti   outside of langpack postinst, that is
06:59   pitti   but we solve the mirror problem, the download redundancy and pretty much all other problems AFAICS
07:00   pitti   actually, when I think about it
07:00   mdz     true, the ideal solution for .deb delta updates in apt wouldn't necessarily work for rsync mirrors
07:00   pitti   the translations don't need to be updated in sync with pacackage updates
07:00   pitti   the user could click onto a button "update translations" which would just call update-language-...
07:01   pitti   especialyl in stable releases this is a fine thing
07:01   pitti   since you don't normally upgrade pacakges anyway
07:01   mdz     this is similar to how the OAV signature stuff works, for example
07:01   pitti   (modulo security udpates)
07:02   pitti   Proposal: I will think about it once more and post my result to the list
07:02   mdz     yes, we need to close this meeting
07:02   pitti   unless anybody has a great new idea
07:02   mdz     interest is waning :-)
07:02   pitti   I'll sum up on the list
07:03   mdz     let's give it some more thought and revisit in Mataro
07:03   pitti   oh right, a BOF!
07:03   Mithrandir      BOFs are good.
07:03   mdz     we will have translators, etc. there as well
07:03   pitti   I will work out the current plan for Mataro
07:03   pitti   mdz: right, I already saw the planned bof on the wiki
07:04   mdz     ok
07:04   mdz     thanks, everyone
07:04   mdz     meeting adjourned
07:04   pitti   byebye

Generated by irclog2html.pl 2.1 by Jeff Waugh - find it at freshmeat.net! 

MeetingLog/Ubuntu/2004-12-11 (last edited 2008-08-06 16:37:49 by localhost)