Contents

  1. Client-side tools
  2. NEW Processing
    1. partner archive
  3. Component Mismatches and Changing Overrides
  4. Removals
    1. Manual
    2. Blacklisting
    3. Removals in Debian
    4. Failed SRUs
  5. Syncs
  6. NBS
  7. i386 whitelist updates
  8. Adjusting Launchpad ACLs
    1. Package sets
  9. Raw UEFI uploads
  10. Useful tools
    1. Archive state checks
    2. NEW handling
    3. Moving Packages to Updates
    4. Publishing security uploads from the ubuntu-security private PPA
    5. Publishing packages from the ubuntu-mozilla-security public PPA
    6. Publishing packages from the ubuntu-security-proposed public PPA to proposed
    7. Copying PPA kernels to proposed
    8. Copying security uploads to updates
    9. Other copies
    10. Handling updates to partner
    11. Diffs for unapproved uploads
  11. Useful runes
    1. reprocess-failed-to-move
    2. Reprocessing failed source uploads
  12. Stable release updates
    1. langpack SRUs
  13. Other archives
  14. Useful web pages
  15. Chroot management
  16. Archive administration shifts
  17. Archive Administration and Freezes
  18. OEM metapackages
  19. Outdated
    1. Logging In

This page details the processes for the Ubuntu Package Archive Administrators team, and hopefully provides a decent guide for new members of the team.

Bugs should be filed against the appropriate packages, and the team subscribed (not assigned) to the bug.

The requests can be found at https://launchpad.net/~ubuntu-archive/+subscribedbugs - the former sometimes fails to load, if in doubt https://bugs.launchpad.net/~ubuntu-archive/+bugs?orderby=-date_last_updated&start=0 also includes assigned bugs but works more reliable..

Team members may assign bugs to themselves and mark them In Progress if they're working on them, or discussing them; to act as a lock on that request.

1. Client-side tools

We are transitioning towards client-side administration as the necessary facilities become available via the Launchpad API. To get hold of these tools:

2. NEW Processing

To work with the upload queue, you may either use the web interface or the queue API client in ubuntu-archive-tools. The API client should generally be faster and more flexible; in particular, it is not currently possible to override individual binaries using the web interface.

Both source packages and new binaries which have not yet been approved are not automatically accepted into the archive, but are instead held for checking and manual acceptance. Once accepted they'll be automatically approved from then on.

The current queue can be obtained with:

This is the NEW queue for the development series of Ubuntu by default; you can change the queue with -Q, the distro with -D and the release using -s. To list the UNAPPROVED queue for ubuntu/xenial, for example:

Packages are placed in the UNAPPROVED queue if they're uploaded to a closed distribution, and are usually security updates or similar; this should be checked with the uploader.

You can give an string argument after info which is interpreted as a substring match filter.

To obtain a report of the size of all the different queues for a particular release:

Back to the NEW queue for now, however. You'll see output that looks somewhat like this:

The number at the start can be used with other commands instead of referring to a package by name. The next field shows you what is actually in the queue, "S-" means it's a new source and "-B" means it's a new binary. You then have the package name, the version and how long it's been in the queue.

New sources need to be checked to make sure they're well packaged, the licence details are correct and permissible for us to redistribute, etc. See Packaging new software, debian/copyright file and Debian's Reject FAQ. You can fetch a package from the queue for manual checking:

Or, if you just want to print the URLs so that you can fetch them on a system with a faster network connection:

The source is now in the current directory and ready for checking. Any problems should result in the rejection of the package (also send a mail to the uploader explaining the reason and Cc ubuntu-archive@lists.ubuntu.com):

If the package is fine, you should next check that it's going to end up in the right part of the archive. On the next line of the info output, you have details about the different parts of the package, including which component, section, etc. it is expected to head into. One of the important jobs is making sure that this information is actually correct through the application of overrides.

To alter the overrides for a package, use:

Where the override can be -c <component>, -x <section>, and/or (for binary packages) -p <priority>. If you want to limit the override application to only source or only binary packages, use the --source or --binary option respectively.

Often a binary will be in the NEW queue because it is a shared library that has changed SONAME. In this case you'll probably want to check the existing overrides to make sure anything new matches. These can be found in /ubuntu/indices on Ubuntu mirrors.

Currently a special case of this are the kernel packages, which change package names with each ABI update and build many distinct binary packages in different sections. A helper tool has been written to apply overrides to the queue based on the packages that are currently published:

Binary packages are not often rejected (they go into a black hole with no automatic notifications), so, check the .deb contains files, run lintian on it and file bugs when things are broken. The binaries also need to be put into universe etc as appropriate even if the source is already there.

If you're happy with a package, and the overrides are correct, accept it with:

You can also use ./queue accept binary-name which will accept it for all architectures.

2.1. partner archive

The Canonical partner archive, though not part of Ubuntu proper, is managed using the same tools and queues. As such, use the same procedures when processing partner packages. E.g. (notice 'Component: partner'):

Use -A ubuntu/partner to remove a package:

The rules governing package inclusion in partner are not the same as those for the main Ubuntu archive. See ExtensionRepositoryPolicy for the Technical Board's requirements regarding the contents of add-on repositories made available through the Software Sources interface.

3. Component Mismatches and Changing Overrides

Sadly packages just don't stay where they're put. SeedManagement details how packages get chosen for the main component, the various meta packages and presence on the CD. What it doesn't point out is that packages which fall out of the seeding process are destined for the universe component.

Every 30 minutes or so, the difference between what the seeds expect to be true and what the archive actually believes is evaluated by the component-mismatches tool, and the output placed at:

This is split into four sections:

Source and binary promotions to main

Binary only promotions to main

Source and binary demotions to universe

Binary only demotions to universe

Once you've determined what overrides need to be changed, use the change-override client-side tool to do it.

To promote a binary package to main:

To demote a source package and all of its binaries to universe:

To demote a binary package to universe to solve a component-mismatch issue (note the -proposed target rather than the release pocket), typically unseeded because the new version introduced an unwanted dependency:

Less-used are the options to just move a source, and leave its binaries where it is (usually just to repair a mistaken forgotten -S):

and the option to move a binary and its source, but leave any other binaries where they are:

4. Removals

4.1. Manual

Sometimes packages just need removing entirely, because they are no longer required. This can be done using the remove-package client-side tool:

By default this removes the named source and binaries, to remove just a binary use -b:

"NBS" is a common short-hand meaning that the binary is No-longer Built by the Source.

To remove just a source, use -S.

The tool tells you what it's going to do, and asks for confirmation before doing it, so it's reasonably safe to get the wrong options and say N.

4.2. Blacklisting

If you remove source packages which are in Debian, and they are not meant to ever come back, add it to the blacklist in lp:~ubuntu-archive/+junk/sync-blacklist, document the reason, and bzr commit it with an appropriate changelog. This will avoid getting the package back to source NEW in the next round of autosyncs from Debian.

4.3. Removals in Debian

From time to time we should remove packages which were removed in Debian, to avoid accumulating cruft and unmaintained packages. This client-side tool (from ubuntu-archive-tools) will interactively go through the removals and ask for confirmation:

Please note that we do need to keep some packages which were removed in Debian (e. g. "firefox", since we did not do the "firefox" → "iceweasel" renaming).

4.4. Failed SRUs

If a package should be removed from -proposed, use the remove-package tool (from ubuntu-archive-tools). e.g., to remove source and binaries for the libreoffice package currently in xenial-proposed:

5. Syncs

Syncing packages with Debian used to be a reasonably common request. It is now self-service for developers using syncpackage. If you are asked to process a sync in the old style, you can use the -s option to syncpackage to do so in the name of the requester.

Most updates available in Debian are automatically synced before DebianImportFreeze, but packages that were previously in Ubuntu will not be automatically reintroduced. To process these interactively:

To sync from any source which is not automatically imported by Launchpad, use the syncpackage tool on a .dsc file. Any uploader can do this; it does not require being an archive administrator.

Likewise, backports are now self-service for members of ubuntu-backporters using backportpackage.

6. NBS

Sometimes binary packages are not built by any source (NBS) any more. This usually happens with library SONAME changes, package renamings, etc. Those need to be removed from the archive from time to time, and right before a release, to ensure that the entire archive can be rebuilt by current sources.

Such packages are detected by archive-cruft-check. This tool does not check for reverse dependencies, though, so you should use checkrdepends -b for checking if it is safe to actually remove NBS packages from the archive.

Look at the half-hourly generated NBS report which shows all NBS packages, their reverse dependencies, and a copy-and-paste-able command to clean up the "safe" ones.

The rest needs to be taken care of by developers, by doing transition uploads for library SONAME changes, updating build dependencies, etc. The remaining files will list all the packages which still need the package in question.

Please refrain from removing NBS kernel packages for old ABIs until debian-installer and the seeds have been updated, otherwise daily builds of alternate and server CDs will be made uninstallable.

7. i386 whitelist updates

The i386 is a partial archive now, details on https://wiki.ubuntu.com/i386

To add packages to the whilelist

Note that if the binary package was built for an earlier release, e.g. oracular for the current plucky, you'll need to adjust the copy-package invocation to ./copy-package -b --from-suite=oracular --to-suite=plucky -e $version $pkg.

the copy should trigger a build on i386, once it's published

8. Adjusting Launchpad ACLs

NOTE: due to bug #562451, archive administrators cannot currently adjust Launchpad ACLs.

The new ArchiveReorganisation brings finer grained access controls than what components can provide. Launchpad ACLs allow individuals and teams to have upload or admin rights on certain packages, referred to as sets. In general, an archive administrator can process requests to create and delete package sets, as well as add or remove packages from package sets. Archive administrators should not add individuals or teams to package sets without explicit TechnicalBoard approval.

8.1. Package sets

Packages can be added to or removed from package sets using the edit-acl tool from ubuntu-archive-tools.

To list the packages currently in the package set mozilla:

To add a package to the mozilla package set:

To remove a package from the mozilla package set:

For more information, please see edit-acl --help.

9. Raw UEFI uploads

Launchpad supports autosigning of EFI binaries using the Secure Boot signature format. This is implemented using a "raw uefi" format upload within a binary package build (see the efilinux package in quantal or later for an example). To provide additional assurance that only trusted EFI bootloaders are signed using this method, packages that include raw UEFI binary uploads land in the unapproved queue and require archive admin approval before they are signed. Archive admins should review the corresponding source upload for correctness prior to approving these uploads.

10. Useful tools

There are other useful tools in your PATH which are invaluable.

10.1. Archive state checks

rmadison (in the devscripts package, run on your client system) examines the current state of the archive for a given binary/source package:

Or when used with -S and a source package, the source and every binary built by it:

checkrdepends lists the reverse dependencies of a given binary:

or source package:

10.2. NEW handling

A lot of churn in NEW comes from Debian imports. Since they already went through NEW in Debian, we should not waste too much time on it, so there are some tools.

10.3. Moving Packages to Updates

Packages in -proposed can be moved to -updates once they are approved by someone from sru-verification, and have passed the minimum aging period of 7 days. Use the sru-release script from ubuntu-archive-tools for this:

Please see --help, you can also use this tool to copy packages to -security and to the current development release. N.B. before copying a package to -security ping a member of the ubuntu-security team.

10.4. Publishing security uploads from the ubuntu-security private PPA

Security uploads in Soyuz are first built, published, and tested in the Security Team's private PPA. To unembargo them, the security team use a tool that re-publishes them to the primary archive. This is now self-service, although you may occasionally be asked to adjust overrides, which can be done in the usual way.

10.5. Publishing packages from the ubuntu-mozilla-security public PPA

Mozilla (ie, firefox and thunderbird) uploads in Soyuz are first built, published, and tested in the Mozilla Security Team's public PPA. To publish them into the main archive, use copy-package from ubuntu-archive-tools. Note that pocket copies to the security pocket should never be done without an explicit request from a member of the Ubuntu Security Team (Mozilla Security Team is not enough), and copies to the proposed pocket should not be done without an explicit request from a member of the SRU Team. Keep in mind that firefox 2.0 and later (ie hardy and later) will always have a corresponding xulrunner package that needs copying.

To publish firefox version 22.0+build2-0ubuntu0.12.04.2 from the ubuntu-mozilla-security PPA to the -security pocket of ubuntu's precise release, you would do the following:

Alternatively, can use lp:ubuntu-qa-tools/security-tools/unembargo like so:

10.6. Publishing packages from the ubuntu-security-proposed public PPA to proposed

This works the same as the ubuntu-mozilla-security PPA, above. Eg, using lp:ubuntu-qa-tools/security-tools/unembargo:

10.7. Copying PPA kernels to proposed

With the new StableReleaseCadence, kernels are built in the kernel team PPA and then pocket copied to the proposed pocket in the main archive once they are ACKd.

The two most useful URLs for this process are the Kernel SRU Report, where you will find links to the tracking bugs (the magnifying glasses), as well as the status of each package in each release, and the Kernel SRU workflow, which outlines in fine detail how each bug task should be handled, what the states mean, etc. The executive summary of the previous link is "only operate on tasks assigned to your team, and only when they're in the Comfirmed state, and when you do, please reassign to yourself and set the task to Fix Released when you're done".

The Pending SRU report has a section for the kernel PPA which shows all newer kernels in the PPA, provides clickable links to open all bugs (with separate CVE bugs), and copy&pasteable copy-proposed-kernel and sru-accept commands (both in ubuntu-archive-tools). To publish linux version 2.6.32-27.49 from the canonical-kernel-team PPA to the -proposed pocket of ubuntu's lucid release, you would do the following:

TODO: A process/script similar to kernel-overrides should be developed to make sure overrides are properly handled for binaries not in main. Is find-bin-overrides from lp:ubuntu-qa-tools good enough?

10.8. Copying security uploads to updates

Security uploads are distributed from a single system, security.ubuntu.com (consisting of one or a small number of machines in the Canonical datacentre). While this ensures much quicker distribution of security updates than is possible from a mirror network, it places a very high load on the machines serving security.ubuntu.com, as well as inflating Canonical's bandwidth expenses very substantially. Every new installation of a stable release of Ubuntu is likely to be shortly followed by downloading all security updates to date, which is a significant ongoing cost.

To mitigate this, we periodically copy security uploads to the -updates pocket, which is distributed via the regular mirror network. (In fact, the pooled packages associated with -security are mirrored too, but mirrored -security entries are not in the default /etc/apt/sources.list to avoid causing even more HTTP requests on every apt-get update.) This is a cheap operation, and has no effect on the timely distribution of security updates, other than to reduce the load on central systems.

The copy-report tool lists all security uploads that need to be copied to -updates. If the package in question is not already in -updates, it can be copied without further checks. Otherwise, copy-report will extract the changelogs (which may take a little while) and confirm that the package in -security is a descendant of the package in -updates. If that is not the case, it will report that the package needs to be merged by hand.

The output of the tool looks like this:

The block of output under "The following packages can be copied safely:" may be copied and pasted in its entirety (or run copy-report --safe). If there is a block headed "The following packages need to be merged by hand:", then make sure that the security team is aware of those cases.

Note that copy-report --safe is run from a cron job as the ubuntu-archive user on ubuntu-archive-toolbox, and so does not typically need to be run by hand.

10.9. Other copies

copy-package is a versatile tool and can be used for a number of other purposes. If you are doing something not documented here, please ask on #ubuntu-release first.

Archive administrators technically (as in, per the ACLs) have the ability to copy into the RELEASE pocket of the current development series. Please do not do this; that pocket is owned by the proposed-migration tools, and those tools have built-in override facilities for the cases where they are wrong. The release team has access to apply such overrides when needed.

10.10. Handling updates to partner

Updates to the partner archive follow a similar process as packages in the Ubuntu archive. Specifically, packages are first built in the proposed pocket, then they need to be copied to the release pocket after they are built and then cleaned up from the proposed pocket. The process is (examples show adobe-flashplugin being updated for xenial):

  1. Review the package in the "unapproved" queue, and if appropriate, accept the package into partner for the proposed pocket (in this example, proposed for xenial)
  2. When the package builds on all architectures, copy it to the partner release pocket (this makes it available to users). Eg: 
    $ ./copy-package -b --auto-approve --from=ubuntu/partner -s xenial-proposed --to-suite xenial adobe-flashplugin

    or to do all at once:

    $ for i in xenial bionic focal groovy; do ./copy-package -b --auto-approve --from=ubuntu/partner -s $i-proposed --to-suite $i adobe-flashplugin ; done
  3. Remove the package from proposed in the partner archive. Eg: 
    $ ./remove-package -A ubuntu/partner -m "copied to release" -s xenial-proposed adobe-flashplugin

    or to do all at once:

    $ for i in xenial bionic focal groovy; do ./remove-package -A ubuntu/partner -m "copied to release" -s $i-proposed adobe-flashplugin ; done

10.11. Diffs for unapproved uploads

The "unapproved" queue holds packages while a release is frozen, i. e. while a milestone or final freeze is in progress, or for post-release updates (like hardy-proposed). Packages in these queues need to be scrutinized before they get accepted.

This can be done with the queuediff tool in ubuntu-archive-tools, which generates a debdiff between the current version in the archive, and the package sitting in the unapproved queue:

-s specifies the release pocket to compare against and defaults to the current development release. Please note that the pocket of the unapproved queue is not checked or regarded; i. e. if there is a hal package waiting in hardy-proposed/unapproved, but the previous version already migrated to hardy-updates, then you need to compare against hardy-updates, not -proposed.

Check --help, the tool has more options, such as specifying a different mirror, or -b to open the referred Launchpad bugs in the webbrowser.

This tool works very fast if the new package does not change the orig.tar.gz, then it only downloads the diff.gz. For native packages or new upstream versions it needs to download both tarballs and run debdiff on them. Thus for large packages you might want to do this manually in the data center.

11. Useful runes

This section contains some copy&paste shell bits which ease recurring jobs.

11.1. reprocess-failed-to-move

In some cases, binary packages fail to move from the incoming queue to the accepted queue. To fix this, run ~lp_buildd/reprocess-failed-to-move as lp_buildd

11.2. Reprocessing failed source uploads

If a source upload fails for spurious/transient reasons (you can check in pepo:/srv/launchpad.net/production-logs/lp_queue/process-upload.log), ask webops to locate the appropriate upload-* directory and corresponding .distro file in /srv/launchpad.net/ubuntu-queue/ (they will probably be in either failed or rejected) and move them both back to /srv/launchpad.net/ubuntu-queue/incoming/.

12. Stable release updates

SRU packages in -proposed must be approved by ~ubuntu-sru before accepting.

Please see https://wiki.ubuntu.com/StableReleaseUpdates#Reviewing_procedure_and_tools

12.1. langpack SRUs

13. Other archives

extras.ubuntu.com is not managed by the Ubuntu archive administration team, but is a PPA owned by the Application Review Board.

14. Useful web pages

Equally useful to the tools are the various auto-generated web pages in ubuntu-archive's public_html that can give you a feel for the state of the archive.

http://people.canonical.com/~ubuntu-archive/component-mismatches.txt

http://people.canonical.com/~ubuntu-archive/germinate-output/

http://people.canonical.com/~ubuntu-archive/priority-mismatches.txt

http://people.canonical.com/~ubuntu-archive/architecture-mismatches.txt

http://people.canonical.com/~ubuntu-archive/testing/xenial_probs.html

http://people.canonical.com/~ubuntu-archive/testing/xenial_outdate.html

http://people.canonical.com/~ubuntu-archive/NBS/ http://people.canonical.com/~ubuntu-archive/nbs.html

15. Chroot management

Warning /!\ Please note that chroot management is something generally handled by Canonical IS (and specifically by Adam Conrad). The following section documents the procedures required should one have to, for instance, remove all the chroots for a certain suite to stop the build queue in its tracks while a breakage is hunted down and fixed, but please don't take this as an open invitation to mess with the buildd chroots willy-nilly.

Soyuz stores one chroot per (suite, architecture).

manage-chroot allows the following actions upon a specified chroot:

Downloading (get) an existing chroot:

The chroot will be downloaded and stored in the local disk name as 'chroot-<DISTRIBUTION>-<SERIES>-<ARCHTAG>.tar.bz2'

Uploading (add/update) a new chroot:

The new chroot will be immediately used for the next build job in the corresponding architecture.

Disabling (remove) an existing chroot:

Warning /!\ Unless you have plans for creating a new chroots from scratch, it's better to download them to disk before the removal (recover is possible, but involves direct DB access)

No builds will be dispatched for architectures with no chroot, the build-farm will continue functional for the rest of the system.

16. Archive administration shifts

This is a next try for establishing Archive Admin shifts during the week. During an AA shift, an archive admin allocates at least 2 hours to work on tasks requiring special ubuntu-archive permissions.

Current members with regular admin shifts are:

On an archive shifts, the following items should be the main focus:

Additionally, the performing the following should be considered:

17. Archive Administration and Freezes

Archive admins should be familiar with the FreezeExceptionProcess, however it is the bug submitter's and sponsor's responsibility to make sure that the process is being followed. Some things to keep in mind for common tasks:

See FreezeExceptionProcess for complete details.

18. OEM metapackages

The archive team runs a script, maintained by the DeveloperMembershipBoard, to automatically grant upload rights to oem-*-meta packages - including packages which aren't yet in Ubuntu (uploaders can upload to NEW for these packages) - to some members of Canonical's OEM delivery team via a packageset called "canonical-oem-metapackages".

The DMB handle applications to the packageset and maintain the code. Our responsibility in the archive team is to run the script reasonably frequently, pull any changes when the DMB ask us to, and arrange for its output to be emailed at least to the devel-permissions list.

See the DMB's knowledge base and the script itself for some further information and links.

19. Outdated

19.1. Logging In

Outdated: this is now on ubuntu-archive-toolbox.internal and works completely differently

A few operations still require direct ssh access to pepo.canonical.com; accounts are provided to some members of the team. (We are transitioning away from direct shell access, so new team members will not get accounts here, thereby encouraging us to complete the transition more quickly.) Changes can only be made as the lp_archive user, to which you'll have sudo access.

So to begin:

The -i is important as lp_archive's .bashrc sets the right environment variables and makes sure the directory with all of the tools is placed in the PATH.

ArchiveAdministration (last edited 2025-03-14 17:08:53 by schopin)