ArchiveReorganisation

Revision 2 as of 2008-06-25 10:11:13

Clear message

THIS IS AN UNAPPROVED DRAFT.

Problem statement

Current archive layout

Since its inception, the Ubuntu archive has been split into four components, organised as follows:

Component

Licensing

Supportedness

Uploader team

main

Free software

Supported

ubuntu-core-dev

restricted

Non-free software (drivers)

Supported

ubuntu-core-dev

universe

Free software

Unsupported

ubuntu-dev

multiverse

Non-free software

Unsupported

ubuntu-dev

This scheme was created before Ubuntu was announced publicly, and before the MOTU team was established to take care of the universe and multiverse components. It also predates the creation of Ubuntu derivatives hosted in the Ubuntu archive, such as Xubuntu and Ubuntu Studio. Of late it has been creaking at the seams. Problems have included:

  • It has been unclear whether packages required for CD images of Ubuntu derivatives should go in main. Our documentation is at best unclear on this.

  • The MOTU team's remit is extremely wide, and is fragmented by needing to cover Ubuntu derivatives as well as general packages untended in other ways.
  • Developers are under pressure to join ubuntu-core-dev in order to upload to relatively small subsets of the archive. For example, developers interested in the use of Ubuntu in education need to join ubuntu-core-dev in order to upload packages on the Ubuntu Education Edition CD images.

  • It is often not clear even to Canonical staff exactly what Canonical supports. Originally this was simply "everything in main", but now we require a distinction between "maintained for security updates" and "commercially supported".

  • Users have very little guidance on what "supported" means, and the available user interfaces only indicate whether a package is maintained for security updates, not whether it is commercially supported (by Canonical or otherwise) or whether it is well-tested with the flavour of Ubuntu they are using (for example, unless they know otherwise, Kubuntu users will likely want to default to using KDE applications rather than GNOME applications).
  • When a package is moved from universe to main, many of the people who previously tended to it lose the ability to upload it.

Much of the metadata needed to solve these problems already exists, but is not exposed to users in any useful form; instead, we simply conflate everything we care about into main, robbing ourselves of the ability to express very much more than a binary state.

Tracking of packages

The contents of Ubuntu installation images, together with the basic data used to divide the archive into supported and unsupported components (henceforth simply main and universe for brevity; please read restricted and multiverse as well where appropriate), are tracked using [:SeedManagement:seeds]. These are formed as lists of package names that are "interesting" in particular categories; for example, an Ubuntu desktop needs to have the gnome-panel package installed. Dependencies and build-dependencies need not be explicitly listed in seeds, because germinate will expand them as necessary.

Conceptually, therefore, the contents of main are defined by the union of the output of germinate for all supported flavours (i.e. Ubuntu, Kubuntu, etc.). However, in practice it is desirable to have an additional review step before allowing packages into main, as described in the MainInclusionProcess. Since this must, by policy, be followed for all new source packages entering main (with occasional exceptions for "trivial" changes, such as source packages being renamed for new versions of a library), it is thus not generally possible for an uploader to introduce unsuitable dependencies into main either unintentionally or maliciously. This system may briefly be characterised as human moderation of access control changes.

Problems with access control

The ubuntu-core-dev and ubuntu-dev teams were originally intended as rungs on a ladder. Contributors would start out by submitting patches until they were deemed capable of uploading a limited set of packages without prior review; once they had accumulated enough experience there, they would proceed to ubuntu-core-dev where they would be able to upload to the entire archive.

Over time, this has proven to be inadequate in various specific ways, though basically sound in general. For example, in order to be able to upload the kernel, kernel developers must spend time developing extensive packaging experience in order to gain ubuntu-dev and then ubuntu-core-dev, despite the fact that their interests or even jobs are not particularly aligned with this. The Technical Board has on occasion worked around this by ad-hoc exceptions on the condition that people ask for review before uploading packages outside their areas of interest, but this is rather opaque and requires a certain degree of persuasion in each individual case. Similarly, Kubuntu developers must generally demonstrate a general ability to understand the Ubuntu system beyond their own flavour in order to gain ubuntu-core-dev, even though without that they could easily be trusted to do sensible things with packages only on the Kubuntu CD images.

Nothing in the following proposal is intended to diminish the rôle of ubuntu-core-dev. That team should still consist of highly experienced and competent developers with a wide-ranging understanding of the Ubuntu system. They will often be called upon to arbitrate among conflicting requirements from different Ubuntu flavours, or to make changes with an understanding of their effect on a variety of different subsystems. However, this is a team where quality rather than quantity is important, and it makes sense both to reduce the pressure on people to join before they are ready in order to get their work done, and to reduce the demands on ubuntu-core-dev members for sponsorship.

Proposal

This proposal is divided into two parts. The first (and simpler) deals with exceptions useful in specific cases; the second deals with wider issues.

Additional uploaders

As of Launchpad 1.2.5 (though there is no UI yet), it is possible to define additional uploaders for an individual package. For small numbers of packages, this process may well be suitable. For example, it is likely to make sense for experienced kernel team members to be able to upload the kernel regardless of whether they have substantial enough packaging experience to gain ubuntu-core-dev privileges. Similarly, when a single package that has been well-cared-for in universe moves to main due to the good support it has been receiving from its de facto maintainer, it is usually desirable for that maintainer to be able to continue their work.

We propose that prospective uploaders apply to the Technical Board for permission, and that such permission should generally be granted based on experience with the package in question more than on experience with the rest of the Ubuntu system. Naturally, some level of familiarity with and respect for Ubuntu processes (such as freeze procedures) should be required. As usual, the Technical Board may delegate this as they think appropriate.

Improved flavour handling

For full-scale Ubuntu derivatives with their own maintenance teams, it is likely that the above process will be inadequate, or at best only workable with a substantial amount of busy-work and automation. Furthermore, it does not solve the problems of expressing support from entities other than Canonical (such as development teams) or of indicating to users which of a set of packages is most likely to work best with the other packages already on their computer.

Good examples of the scaling problems here are Ubuntu Studio and Mythbuntu. Ubuntu Studio has 141 unique source packages in Ubuntu 8.04 LTS (i.e. packages that are part of Ubuntu Studio, but part of no other "set"). Mythbuntu has 124. Kubuntu and the KDE4 remix have 115 between them. Xubuntu has 65. In each case, it makes complete sense for each development group to be able to work on their set of packages, and neither the MOTU community nor the core developer community are likely to be as qualified to do so simply because they largely do not use them directly. Maintaining independent access control lists for each of these would quickly spiral out of control.

However, it is clearly necessary to put measures in place to stop people gaming the system. That is, simply adding a dependency to a package you control should not allow you to upload that other package, or to take control of it from somebody else.

Fortunately, identical measures are already in place for the distinction between main and universe, as described above. Adding a dependency to a package in main does not automatically cause the dependency to propagate to main as well; an archive administrator has to acknowledge it and change the overrides.

We propose that the existing mechanisms for archive administrators to apply overrides to packages placing them in the main or universe component should be replaced with mechanisms for them to assign them to one or more uploader teams. In parallel, the tools that monitor override discrepancies should be replaced with similar tools to monitor discrepancies in assigned uploader teams. The rules should be as follows:

  • Packages in the "platform" seed collection (perhaps to be renamed "core") may be uploaded only by ubuntu-core-dev.

  • Packages in a single other seed collection may be uploaded only by ubuntu-core-dev or the team responsible for that seed collection (normally, the same team that can commit to the seeds).

  • Packages in more than one other seed collection may be uploaded only by ubuntu-core-dev or any of the responsible teams. However, archive administrators are expected to investigate packages that want to move into this state, and may consider moving them into the "platform" collection if they are in fact important central packages on which many other things depend.

  • Unseeded packages may be uploaded only by ubuntu-core-dev or ubuntu-dev (i.e. MOTU). The MOTU community as a whole will be responsible for taking care of packages that are not under the care of an individual flavour maintenance team, although we expect that many individual MOTU members will also be involved with flavour maintenance teams.

The initial set of recognised flavours will be: Ubuntu (desktop), Ubuntu (server), Kubuntu, Ubuntu Education Edition (né Edubuntu), Ubuntu Mobile, Xubuntu, Mythbuntu, and Ubuntu Studio. The addition of further flavours will require the authorisation of the Technical Board, as that amounts to a mass delegation of the ability to confer upload authority.