Migrating packages from -proposed to release

Uploads to the Ubuntu development series are directed to the -proposed pocket, and must build there and pass certain tests before they are promoted to the release pocket for general use. This process is derived from that used by Debian to maintain the testing suite, but there are a few Ubuntu-specific differences. Although the observant will note that this document is also derived from Debian's documentation, it is intended to be a self-contained description for Ubuntu developers.

One important difference that those familiar with the Debian process should be aware of is that -proposed is a partial suite: that is, it only contains packages that have not yet migrated, and must be used on top of the release pocket. This can cause some confusion where removals are concerned. Furthermore, the Ubuntu process has no arbitrary delays and no consideration of release-critical bugs, so there is very little benefit to users in using -proposed and such use is strongly discouraged. (On the other hand, the official build infrastructure builds packages against -proposed, and this is normally also a good idea when test-building packages locally.)

A (particular version of a) package will move into the release pocket when it satisfies all of the following criteria:

1. It must be built and published on all architectures it is currently published for in the release pocket (this applies to each binary, but it is acceptable if other binaries from the same source for the same architecture are up to date in -proposed and the binary in question has been removed);

2. All of its dependencies must either be satisfiable by packages already in the release pocket, or be satisfiable by the group of packages which are going to be installed at the same time;

3. If the package defines any autopkgtest test suites, they must pass;

4. The operation of installing the package into the release pocket must not break any packages currently in the release pocket.

A package which satisfies the first three of the above is said to be a "Valid Candidate".

The update script shows when each package might move from -proposed into the release pocket. The output is twofold:

• The update excuses: list of all candidate package versions and the basic status of their propagation into the release pocket; this is the first stage of migration, and only packages that pass it proceed to

• The update output: the complete, rather crude output of the proposed-migration scripts as they recurse through the candidates.

Frequently Asked/Answered Questions

How could installing a package into the release pocket possibly break other packages?

The structure of the distribution archives is such that they can only contain one version of a package; a package is defined by its name. So, when the source package acmefoo is installed into the release pocket, along with its binary packages acme-foo-bin, acme-bar-bin, libacme-foo1 and libacme-foo-dev, the old version is removed.

However, the old version may have provided a binary package with an old soname of a library, such as libacme-foo0. Removing the old acmefoo will remove libacme-foo0, which will break any packages which depend on it.

Evidently, this mainly affects packages which provide changing sets of binary packages in different versions (in turn, mainly libraries). However, it will also affect packages upon which versioned dependencies have been declared of the ==, <= or << varieties.

When the set of binary packages provided by a source package change in this way, all the packages that depended on the old binaries will have to be updated to depend on the new binaries instead. Because installing such a source package into the release pocket breaks all the packages that depended on it in the release pocket, some care now has to be taken: all the depending packages must be updated and ready to be installed themselves so that they won't be broken. In some cases, manual intervention by the release team may be required.

If you are having problems with complicated groups of packages like this, contact the ubuntu-devel mailing list or the #ubuntu-release IRC channel for help.

I still don't understand! The proposed-migration scripts say that this package is a valid candidate, but it still hasn't gone into the release pocket.

This tends to happen when in some way, directly or indirectly, installing the package will break some other package.

Remember to consider your package's dependencies. Suppose your package depends on libtool, or libltdlX. Your package won't go into the release pocket until the right version of libtool is ready to go in with it.

In turn, that won't happen until installing libtool doesn't break things already in the release pocket. In other words, until all other packages which depend on libltdlY (where Y is the earlier version) have been recompiled, and all their autopkgtests pass, etc, none of these packages will enter the release pocket.

This is where the textual output is useful: it gives hints (albeit very terse ones) as to which packages break when a valid candidate is added to the release pocket.

My package is stalled because it's out of date on some architecture. What do I do?

Check the status of your package in Launchpad. If the package doesn't build, it will be marked as failed; investigate the build logs and fix any of the problems that are caused by your package's sources.

If you happen to notice that some architectures have built the new version of your package, but it isn't showing up in the output of the proposed-migration scripts, then you just have to be a bit more patient until the publisher runs. This should happen well within half an hour.

If you have explicitly dropped the architecture from the Architecture list in the control file, and the package has been built for that architecture before, you will need to request that the old binary package for this architecture be removed from the archive before your package can transition to the release pocket. You need to file a bug on your package requesting removal of the dropped architecture's packages, and subscribe the ubuntu-archive team.

Are there any exceptions? I'm sure acmefoo has just made it into the release pocket despite not satisfying all of the requirements.

The release team can override the rules in a few ways:

• They can decide that the breakage caused by the installation of a new library will make things better rather than worse, and let it go in along with its flotilla of dependents.
• They can also manually remove packages from the release pocket that would be broken, so that new stuff can be installed.
• They can force autopkgtest failures to be ignored.

Can you provide a real, non-trivial example?

Here's one: when the source package apache is installed into the release pocket, along with its binary packages apache, apache-common, apache-dev and apache-doc, the old version is removed.

However, all Apache module packages depend on apache-common (>= something), apache-common (<< something), so this change breaks all of those dependencies. Consequently, all Apache modules need to be recompiled against the new version of Apache in order for the release pocket to be updated.

Let's elaborate on this a bit further: after all of the modules have been updated in unstable to work with a new Apache, the proposed-migration scripts try apache-common and find out that it breaks all the Apache modules because they have Depends: apache-common (<< the current version), and then try libapache-foo to find out that it doesn't install because it has Depends: apache-common (>= the new version).

However, later they'll apply a different logic (sometimes prompted by a manual intervention): they'll ignore the fact apache-common breaks stuff, and keep going with things that work; if it still doesn't work after we've done everything we can, too bad, but maybe it will work. Later they'll try all the random libapache-foo packages and see that they indeed work.

After everything's been tried, they check how many packages have been broken, work out if that's better or worse than what there was originally and either accept everything or forget about it. You'll see this in update_output.txt on "recur:" lines.

For example:

   recur: [foo bar] baz

basically says "having already found that foo and bar make things better, I'm now trying baz to see what happens, even though that breaks things". The lines of update_output.txt that start with "accepted" indicate things that appear to make things better, and "skipped" lines make things worse.

The update_output.txt file is completely unreadable!

That is not a question.

Let's take an example:

 skipped: cln (0) (150+4)
     got: 167+0: a-40:a-33:h-49:i-45
     * i386: ginac-cint, libginac-dev

This means that if cln goes into the release pocket, ginac-cint and libginac-dev become uninstallable in the release pocket on i386. Note that the architectures are checked in a fixed order and only the problems on the first architecture with problems are shown - that's why the i386 architecture is shown so often.

The "got" line includes the number of problems in the release pocket on the different architectures (until the first architecture where a problem is found — see above). The "i-45" means that if cln would go into the release pocket, there would be 45 uninstallable packages on i386. Some of the entries above and below cln show there were 43 uninstallable packages in the release pocket on i386 at the time.

The "skipped: cln (0) (150+4)" line means that there are still 150 packages to go through after this package until this check of all packages is completed, and that 4 packages have been found already that won't be planned to be upgraded because they would break dependencies. The "(0)" is irrelevant; you can safely ignore it.

Note that there are several checks of all packages in one proposed-migration script run.

Jules Bean initially assembled the frequently asked questions and answers. Colin Watson modified them for Ubuntu.

Additional information

The proposed-migration code is spread across britney1-ubuntu and britney2-ubuntu.

Anthony Towns takes credit for the original implementation of testing in Debian. Colin Watson is at fault for most of the Ubuntu adaptations.