Ubuntu Wiki

This page describes the publication process for packages when it deviates from SecurityTeam/UpdateProcedures.

Kernel

Patching

Patching and testing the kernel is primarily the responsibility of the Ubuntu Kernel team. Tracking kernel CVEs, building patched kernels and publishing those kernels is the responsibility of the Ubuntu Security team. As such, the Ubuntu Security team should:

• Enter kernel CVEs into the Ubuntu CVE Tracker
• Forward this information to the kernel team
• Coordinate the timing of kernel security updates (usually monthly, unless a high priority CVE warrants an earlier date)
• Coordinate the Ubuntu Kernel team's work with other vendors as appropriate

Building

Once the kernel team is satisfied with their patching and testing, they will provide packages on chinstrap.canonical.com, currently in chinstrap:~smb/security/srcpkg. Since the kernels are large, the packages should be remotely signed and uploaded from chinstrap (see below). To verify, sign and upload:

1. On chinstrap, copy the kernel team's packages to ~/sign:

   $ cd ~smb/security/srcpkg/ # requires you are in the 'kernel_devs' group
   $ test -d ~/sign/ || mkdir -m 0750 ~/sign/ ; chgrp ubuntu_security ~/sign/
   $ cp * ~/sign/

2. On local system (may require setup, see below), verify and sign the packages in ~/sign on chinstrap:

   $ $UST/package-tools/u-verify-chinstrap      # verify the signatures in ~/sign
   $ $UST/package-tools/u-sign-chinstrap        # sign the packages in ~/sign
   $ $UST/package-tools/u-verify-chinstrap      # reverify the signatures in ~/sign

3. If needed, on chinstrap setup the kern-up symlink:

   $ test -e ~/bin/kern-up || ln -s /home/kees/bin/kern-up ~/bin/kern-up

4. On chinstrap, perform a test upload with kern-up. Eg:

   $ cd ~/sign
   $ ~/bin/kern-up # | sed 's/ security\-/ security-proposed-/' # for proposed ppa
   dput security-dapper linux-source-2.6.15_2.6.15-55.87_source.changes
   dput security-hardy linux_2.6.24-28.75_source.changes
   dput security-jaunty linux_2.6.28-19.64_source.changes
   dput security-karmic linux_2.6.31-22.63_source.changes
   dput security-karmic linux-mvl-dove_2.6.31-214.30_source.changes
   dput security-karmic linux-ec2_2.6.31-307.17_source.changes
   dput security-lucid linux_2.6.32-24.41_source.changes
   dput security-lucid linux-mvl-dove_2.6.32-208.24_source.changes
   dput security-lucid linux-meta-mvl-dove_2.6.32.208.11_source.changes
   dput security-lucid linux-ec2_2.6.32-308.15_source.changes
   dput security-lucid linux-ti-omap_2.6.33-502.10_source.changes
   dput security-lucid linux-fsl-imx51_2.6.31-608.19_source.changes

   Compare the output of kern-up with Kernel/Dev/ABIPackages. Ignore the netbook kernels cause they are outside the archive. Also, linux-qcm-msm/lucid is abandoned. If there is an ABI bump, then the ABI meta source package is also listed, otherwise it is not. Every "topic branch" (i.e. source package and referred to as 'git branch' in Kernel/Dev/ABIPackages) has up to two "meta" packages that define ABI, but normally there is just one. Sometimes there is an additional "ports" meta for the non-supported archs. Kernel/Dev/ABIPackages will always have the most up to date information, so consult it with each update (kern-up may need to be adjusted if the kernel team makes changes).

5. Compare the ABIs of the packages output by kern-up with the archive. If there is an ABI bump and the meta package is missing, contact the kernel team.

6. On chinstrap, upload the kernels (see the 'Setup' section below if publishing for the first time):

   • If non-ABI bump, do ~/bin/kern-up --real

   • If ABI bump:

     1. Take the output of ~/bin/kern-up and run the individual dput commands for each kernel and meta package, being careful to not upload any ABI-tracking packages at this time

     2. Wait for the kernels to build on all architectures

     3. After the kernels are finished building, for each of the remaining ABI-tracking packages (as seen in the output of ~/bin/kern-up), run the dput commands for that package

   • If destined for the ubuntu-security-proposed PPA, take the output of ~/bin/kern-up from above (after uncommenting the pipe to sed) and run the individual dput commands

7. On chinstrap, verify all the packages were uploaded by comparing the number of .source.changes files with the number of .upload files in the ~/sign directory:

   $ ls -1 ~sign/*_source.changes | wc -l
   $ ls -1 ~sign/*.upload | wc -l

Testing

Most testing is performed by the kernel team. The Ubuntu Security should at a minimum do the following:

• Using copy_sppa_to_repos from UST, copy the kernels to your local repository. Please see the instructions at the top of copy_sppa_to_repos for different kernels, ABI-tracking packages and meta-packages.

• Using the meta packages, perform upgrade testing for all affected releases for both i386 and amd64. This can be done by ensuring linux-image-generic (linux-image-amd64-generic or linux-image-386 for Dapper) is installed, then performing an apt-get dist-upgrade to pull in the packages from your local repository.

• After upgrading, verify the following:

  $ uname -a
  $ cat /proc/version_signature    # for non-Dapper

• Verify the QRT test scripts for the kernel pass for both i386 and amd64. Run all $QRT/scripts/test-kernel*py scripts except test-kernel-hardening.py (as a convenience, $QRT/notes_testing/kernel/kernel-test-wrapper.sh can be used to automate these first 3 steps)

• Log into an Ubuntu desktop with the new kernel, and verify the basic desktop works (mouse, keyboard, display, networking and creating/editing a file)

If there are reproducers or test cases, try to forward them to the kernel team (or better yet, integrate them into QRT before they do their testing). Private reproducers will need to be tested by the Ubuntu Security team. When possible, include a regression test for the patched functionality along with the test to see if the bug is fixed (ie, "Did this fix the bug? Did this introduce a regression?"). It is probably a good idea to adjust test_updated_modules() in $QRT/scripts/test-kernel-root-ops.py for any modules that have been updated (this will perform a modinfo, modprobe and rmmod on the module).

Finally, using virtualization for testing is fine most of the time, but if the patch is for a problem with real hardware, every effort should be made to test the patch on that hardware.

Publishing

In general, publication is the same as with other security updates. Keep in mind the following:

• Unembargo all non-meta packages at the same time, then after they are mirrored to security.ubuntu.com, upload the meta packages. This will ensure that people don't get a meta package that depends on a kernel that doesn't exist.

• While not required, you can use the pull-usn-desc.py tool from UCT. This is helpful since kernel updates typically have many CVEs to describe in the USN. You give it the CVE list that is in new-usn.sh and it will output example text that you can paste into new-usn.sh and edit. Eg:

  $ cd $UCT
  $ $UCT/scripts/pull-usn-desc.py --cve CVE-... --cve CVE-...

• The title for the USN should be 'Linux kernel vulnerabilities'
• The summary for the USN should be something like 'linux, linux-{ec2,fsl-imx51,mvl-dove,source-2.6.15,ti-omap} vulnerabilities'
• In the minimum binaries section of the USN, use linux-image*, omitting debug and dbgsym packages

ABI bump for security and updates

TODO

ABI bump for security only

When a -proposed kernel has an ABI bump and makes it into -updates, then the next security update kernel will be an ABI bump for -security only users (since security fixes pull from -updates). Since this is an ABI bump for -security only users, the ABI meta source packages and ABI-tracking source packages must be copied from -updates to -security after all of the -security kernels are mirrored. Look at Kernel/Dev/ABIPackages for the list of packages to copy over. For example, if we have a security update for the 'master' kernel (ie, not arm, not backports, etc) and lucid-security currently has 2.6.32-25.45, lucid-updates has 2.6.32-26.47 and the pending lucid-security update has 2.6.32-26.48, then:

• Unembargo the pending lucid-security kernel as normal
• Wait for it to be fully mirrored to security.ubuntu.com

• Review Kernel/Dev/ABIPackages. In this example, the affected ABI meta and meta-tracking packages for the 'master' kernel are linux-meta, linux-ports-meta and linux-backports-modules-2.6.32.

• Have an archive admin copy the packages from -updates. In this example:

  $ copy-package.py -vbs lucid-updates --to-suite=lucid-security linux-meta
  $ copy-package.py -vbs lucid-updates --to-suite=lucid-security linux-ports-meta
  $ copy-package.py -vbs lucid-updates --to-suite=lucid-security linux-backports-modules-2.6.32

2.6.32-26.48

Mozilla

Patching and Building

Patching and building is currently the responsibility of the ubuntu-mozillateam, specifically ChrisCoulson. Mozilla products have a standing MicroReleaseException and so the ubuntu-mozillateam will get official tarballs from upstream, add/update the debian/ directory and push to the ubuntu-mozilla-security PPA. The ubuntu-mozillateam will also ask for USN to put in the changelog prior to preparing updates since upstream does not make public security vulnerabilities prior to release. People reading the changelog are then able to see the USN and look up the details online. Once packages are built, you can use the standard Ubuntu Security team's tools for publication (using the --ppa option where appropriate).

Testing

Verify the QRT test-browser.py script for the all affected products passes for both i386 and amd64. This script will test a variety of functions, test pages, SSL, javascript, plugins, etc. Since some packages other than Firefox use XUL or NSS, the following gives information on basic testing procedures when a particular source package is updated. These instructions should provide good enough test coverage for a particular update, but are not intended to provide exhaustive testing procedures.

Ubuntu 8.04 LTS

1. this will test java, flash, totem, etc so those applications don't have to be tested separately, but be sure to test both icedtea6-plugin and sun-java6-plugin
2. many packages use NSS, but testing the additional affected binaries should provide good test coverage

Ubuntu 9.04

1. this will test java, flash, totem, etc so those applications don't have to be tested separately, but be sure to test both icedtea6-plugin and sun-java6-plugin
2. many packages use NSS, but testing the additional affected binaries should provide good test coverage

Ubuntu 9.10

1. be sure there are no new AppArmor denials after running the script

2. this will test java, flash, totem, etc so those applications don't have to be tested separately, but be sure to test both icedtea6-plugin and sun-java6-plugin. Also
3. many packages use NSS, but testing the additional affected binaries should provide good test coverage

Ubuntu 10.04 LTS

1. be sure there are no new AppArmor denials after running the script

2. this will test java, flash, totem, etc so those applications don't have to be tested separately, but be sure to test both icedtea6-plugin and sun-java6-plugin. Also
3. many packages use NSS, but testing the additional affected binaries should provide good test coverage

Ubuntu 10.10

1. be sure there are no new AppArmor denials after running the script

2. this will test java, flash, totem, etc so those applications don't have to be tested separately, but be sure to test both icedtea6-plugin and sun-java6-plugin. Also
3. many packages use NSS, but testing the additional affected binaries should provide good test coverage

The above assumes the firefox 3.6 transition is complete. More testing procedures can be found in Testing/Firefox3.6.4Upgrade/SecurityPublication.

Publishing

The publication procedure is essentially the same as for regular security updates except:

• The packages live in the ubuntu-mozilla-security PPA. When calling sis-changes or unembargo.py, you must use --ppa=ubuntu-mozilla-security

• Because of limitations in LP, copying from the ubuntu-mozilla-security puts everything into main, so you will need a member of ubuntu-archive to adjust the overrides as documented in ArchiveAdministration

• The USN is preallocated, so you must look in the debian/changelog for the USN instead of assigning a new one

Chromium Browser

chromium packages are currently community supported and provided by Fabien Tassin. He will normally announce an update is available and put the packages somewhere in http://people.ubuntu.com/~fta/chromium/. These packages are large, so sponsoring should be:

1. On your local machine, get the new sources with: wget --mirror -np http://people.ubuntu.com/~fta/chromium

2. On your local machine, compare the new sources with the the last published sources and verify the new packages follow MicroReleaseExceptions

3. Once the packages are verified as ok, copy them to the ~/sign directory on chinstrap (you can wget --mirror -np these from chinstrap)

4. Verify and sign the packages from you local machine (may require setup, see below):

   $ $UST/package-tools/u-verify-chinstrap      # verify the signatures in ~/sign
   $ $UST/package-tools/u-sign-chinstrap        # sign the packages in ~/sign
   $ $UST/package-tools/u-verify-chinstrap      # reverify the signatures in ~/sign

5. dput from chinstrap (may require setup, see below):

   $ dput security-proposed-lucid chromium-browser_*.10.04.1_source.changes
   $ dput security-proposed-maverick chromium-browser_*.10.10.1_source.changes

Rather than uploading directly to the security PPA, we instead basically use the Ubuntu Security team's sponsored upload procedures:

• Build in ubuntu-security-proposed

• Once done building, pocket copy them to -proposed and update the bug in Launchpad

• Verify the packages in -proposed using (at least) lp:qa-regression-testing/scripts/test-browser.py and document it in the bug

• Pocket copy from -proposed to both -security and -updates. Unlike other packages in -proposed, these do not have to wait 7 days to be pocket copied (because it has an MRE), but part of the condition of the MRE is that the testing must be documented in a bug.

Sponsoring on chinstrap

Sometimes developers or teams will provide packages on chinstrap.canonical.com. When these source packages are large, the packages can be remotely signed and uploaded from chinstrap (you do not need nor should have your ~/.gnupg on chinstrap!). The basic process is:

1. If it doesn't already exist, create the ~/sign directory on chinstrap with:

   $ test -d ~/sign/ || mkdir -m 0750 ~/sign/ ; chgrp ubuntu_security ~/sign/

2. Copy the packages to sponsor into the ~/sign directory

3. On local system (may require setup, see below), verify and sign the packages in ~/sign on chinstrap:

   $ $UST/package-tools/u-verify-chinstrap      # verify the signatures in ~/sign
   $ $UST/package-tools/u-sign-chinstrap        # sign the packages in ~/sign
   $ $UST/package-tools/u-verify-chinstrap      # reverify the signatures in ~/sign

4. dput the packages from chinstrap (may require setup, see below)

Setup

• In order to upload packages from chinstrap, you need to either use '--unchecked' with dput or import your signing key into your keyring on chinstrap. Since you do not want your actual keys there, we can create a keyring specifically for chinstrap. On your local machine:

  $ mkdir -m 700 ~/gnupg.chinstrap
  $ gpg --keyserver keyserver.ubuntu.com --homedir=~/gnupg.chinstrap --recv-keys 0x<your key>
  $ scp -pr ~/gnupg.chinstrap/ chinstrap.canonical.com:~<your username on chinstrap>/.gnupg

  On chinstrap, be sure to check ~/.gnupg so that you have imported the key and that you didn't accidentally copy over the wrong files:

  $ gpg --list-keys
  $ test ! -s ~/.gnupg/secring.gpg || echo 'WARNING!!! secring.gpg is not empty'

• Depending on how your ssh_config is setup on your local machine, u-sign-chinstrap and u-verify-chinstrap may not work immediately. However, you can create these symlinks and it should work ok:

  $ ln -s $UST/package-tools/u-sign-chinstrap ~/bin/u-sign-chinstrap.canonical.com
  $ ln -s $UST/package-tools/u-verify-chinstrap ~/bin/u-verify-chinstrap.canonical.com

  Now instead of using $UST/package-tools/u-sign-chinstrap, just use ~/bin/u-sign-chinstrap.canonical.com

• On chinstrap, setup your ~/.dput.cf file as in SecurityTeam/UpdateProcedures. Eg:

  [security-maverick]
  fqdn = ppa.launchpad.net
  incoming = ~ubuntu-security/ubuntu/maverick
  login = anonymous
  
  [security-lucid]
  fqdn = ppa.launchpad.net
  incoming = ~ubuntu-security/ubuntu/lucid
  login = anonymous
  
  [security-karmic]
  fqdn = ppa.launchpad.net
  incoming = ~ubuntu-security/ubuntu/karmic
  login = anonymous
  
  [security-hardy]
  fqdn = ppa.launchpad.net
  incoming = ~ubuntu-security/ubuntu/hardy
  login = anonymous
  
  [security-dapper]
  fqdn = ppa.launchpad.net
  incoming = ~ubuntu-security/ubuntu/dapper
  login = anonymous