Publishing an Update to Launchpad
If also unembargoing a development release from the Ubuntu Security PPA, unembargo the development release first (see below).
Prepare your local configuration
Upload the updated source packages via dput to each release's Security PPA target. Notification about failed builds should be automatically sent to email@example.com.
Note for cloud images, LXD images, Ubuntu on Windows images, cloud archive, Ubuntu Core and/or Ubuntu Touch:
If the update requires urgently updating the cloud archive, alert jamespage that an update is coming that requires a copy from the security pocket to the cloud archive. A list of packages in each cloud archive release can be found at http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/ (Actual repository is located here: http://ubuntu-cloud.archive.canonical.com/ )
Ubuntu Core stable images/OS and kernel snaps are QA'd (snapd SRU's have similar QA) and released on a two-week cadence and therefore will bundle security updates every two weeks using a manual process (currently) that is managed by the Ubuntu Core team. Out-of-cadence emergency updates may be performed for urgent issues. If the update requires urgently updating Ubuntu Core, alert JamieBennett, mvo, and olli that an update is coming that requires an emergency update.
Ubuntu Touch images are updated via OTA using a six week cadence. Security updates automatically flow from the archive into rc-proposed daily if the updated package doesn't exist in the stable-phone-overlay ppa. If it does exist in the stable-phone-overlay, the package in the stable-phone-overlay PPA will need to be updated via the normal silo process. In this manner, security updates are bundled together every 6 weeks. Out-of-cadence emergency updates may be performed for urgent issues. If the update requires urgently updating the Ubuntu Touch stable images, alert pmcgowan and sil2100 that an update is coming that requires an emergency update.
Set the name of the source package being updated: export SRCPKG="srcpkg1 srcpkg2..."
Wait for the finished builds on all supported architectures to finish and appear at the Ubuntu Security PPA: $UCT/scripts/sis-changes --action check-build $SRCPKG
NOTE: For ESM: $UCT/scripts/sis-changes --action check-build $SRCPKG --ppa ubuntu-security/esm --include-eol
Unembargo by copying the packages into the archive accepted queue with: $UQT/security-tools/unembargo $SRCPKG
NOTE: For ESM: skip this step.
If this is a sponsored update (for universe), edit the CVE entries in UCT to reflect the update. Update the Security Team Meeting page to credit the person who provided the debdiff.
In general, security updates are not under embargo and can be uploaded directly to the development release without a USN. For updates that are embargoed in the development release, the process is similar to the above, but different due to LP: #334858.
Prepare your local configuration
Set the name of the source package being updated: export SRCPKG="srcpkg1 srcpkg2..."
Upload the updated source packages via dput to the development release's Security PPA target. The distribution name should not contain the '-security' suffix. Notification about failed builds should be automatically sent to firstname.lastname@example.org.
Wait for the finished builds on all supported architectures to finish and appear at the Ubuntu Security PPA: $UCT/scripts/sis-changes --include-devel -r <development release> --action check-build $SRCPKG. NOTE: due to a bug in sis-changes, this doesn't actually work so you'll have to check the PPA manually.
When the issue is public, ask an archive admin to unembargo the package using $UQT/security-tools/unembargo $SRCPKG. Eg, to unembargo curl on utopic, ask an archive admin to use:
$ $UQT/security-tools/unembargo --ppa=ubuntu-security -r utopic --pocket=proposed curl
In theory, the copy-package can also be used by an archive admin, followed by any override adjustments (you can use the find-bin-overrides script from lp:ubuntu-qa-tools/security-tools for a list of overrides). However, this tool may not work with the private ubuntu-security PPA. Eg, to unembargo sudo 1.7.2p7-1ubuntu2 on utopic, you would ask the archive admin to run the following (see ArchiveAdministration for more information):
$ copy-package -b --ppa=ubuntu-security -s utopic --to-primary --to-suite utopic-proposed -e 1.7.2p7-1ubuntu2 sudo $ change-override -c universe -s utopic-proposed sudo-ldap
This may need to be done by a someone who is both a member of ubuntu-security and ubuntu-archive
Delete SRCPKG from Security PPA for the development release. This should be done before unembargoing the stable releases to ensure it is copied only to the release pocket, and not the security pocket for the development release.
- At this point, the publication for the development release is completed (a USN is not issued for the development release). Proceed to unembargo the stable releases as needed (see above).
Occasionally (usually only in the case of an emergency) the Ubuntu Security team may be asked to update or sponsor a package in the Canonical partner archive. After verifying that the source packaging is correct for partner, uploading is a simple matter of dputting the package to the Ubuntu archive (the Section specified in the packaging tells Launchpad to put it in partner). Launchpad restricts copies from PPAs to the partner archive, so the packages must be uploaded directly to the archive (requires membership in canonical-partner-dev).
Prepare your local configuration
Set the name of the source package being updated, using the ABI Packages page as a guide (git branches with stars are not handled by the Security Team): export SRCPKG="srcpkg1 srcpkg2..."
Make sure to check the ABI is consistent: $UQT/security-tools/kernel-abi-check
- This will say if the ABI has changed, and what the version was in the last published USN
Make sure you have the latest USN database: cd $UCT; ./scripts/fetch-db database.pickle.bz2
Make sure all the packages are in the archive: $UCT/scripts/sis-changes --action check-build $SRCPKG --ppa ubuntu --pocket Security -r REL
(for main/restricted publications in stable releases)
If applicable, verify that the Certified Public Cloud, Cloud Archive, Ubuntu Core and/or Ubuntu Touch teams have acknowledged the urgent update notification (see 'Note for cloud images, cloud archive, Ubuntu Core and/or Ubuntu Touch')
- Assign a USN (format is NNN-S, and the following instructions assume $USN has been set as desired):
For an old issue that needs correction or continuation, start with the issue's original USN, keep NNN and increase S. (e.g. original issue was 42-1, updated USN will be 42-2).
- For a new issue that affects different software with identical CVEs, get a new USN normally for the first source package, and then keep NNN and increase S for each additional source package. (e.g. CVE-2008-1693 affected both poppler and koffice, so 603-1 was used for poppler and 603-2 was used for koffice). Please note that different versions of the same software (e.g. emacs21 and emacs22) should not do this, but instead use a single USN with S=1.
To create the USN template script, run: $UCT/scripts/sis-changes --download /tmp/pending $SRCPKG && cd /tmp/pending && $UCT/scripts/sis-generate-usn $USN *.changes > ~/new-usn.sh
For announcing kernels that have been already released, add --kernel-mode --filter-bins '^linux-image-\d' --no-new-warn --ppa ubuntu --pocket Security -r REL[,REL] to the sis-changes command line. --no-new-warn is only necessary when publishing a new source tarball
For announcing Mozilla updates, see https://wiki.ubuntu.com/SecurityTeam/PublicationNotes#Mozilla%20Publishing
Edit ~/new-usn.sh to include a correct title, summary, action, description, and then limit the binary list to only those affected by the USN (umt compare-bin may be helpful in determing the affected binaries). Leave all URLs as-is. Have the description proofread by somebody else.
If the update is a regression fix or does not fix CVEs, be sure to create a tracking bug and have new-usn.sh reference at least this one bug (eg, something like 'usn.py $DB $USN --cve "https://launchpad.net/bugs/NNNNNNN"'
- If the update is a regression fix, be sure to replace "vulnerability" with "regression" in the title
- Sometimes the "Software Description" field in the generated USN is unhelpful or inaccurate. In those situations, you'll need to update the description_overrides dict in $UCT/scripts/cve_lib.py with a new description.
To populate the USN database with the new USN details and generate the template email (sent to email@example.com), run: bash ~/new-usn.sh
Wait until the packages are actually mirrored to the archive; the publisher runs hourly at :03, and updates should usually appear on security.ubuntu.com within 20-40 minutes, depending on the size and number of binary packages (note 0403 UTC publication run is skipped due to the Contents generation job). If it is more than 3 hours before the packages are mirrored, then ask #is. To verify that the packages have arrived, run: ssh people.canonical.com "~ubuntu-security/bin/check-upload $USN"
Once packages are in the archive and you are satisfied with the email text (from new-ush.sh), copy the updated USN database by running: ssh people.canonical.com "~ubuntu-security/bin/push-usn-db"
Create a new USN page for this by running: w3m -dump http://www.ubuntu.com/usn/update/usn-$USN; for i in $(seq 1 4); do w3m -dump http://www.ubuntu.com/usn/usn-$USN | awk '/Ubuntu Security/,/Back to top$/' ; done This should return an 'Ok' message (eg, 'Loading USN-1110-1: Ok') followed by a text dump of the page. If this returns 'No such USN: usn-xxx', wait a couple of minutes and try again. Bugs in the microsite can be filed at https://bugs.launchpad.net/usn-website. Urgent issues for problems that are identified as problems with the USN microsite should be directed to mars (coding errors) or #webops (all other issues).
- Once the USN is available on the USN website, GPG sign and send the USN email with the following headers:
To: firstname.lastname@example.org Reply-to: Ubuntu Security <email@example.com> Mail-Followup-to: Ubuntu Security <firstname.lastname@example.org>This can be done any number of ways. Typically it involves forwarding the email inline, adjusting the From to be from you, adjusting the Subject to remove any Fwd text, removing any attachments, adjusting the body text to remove forwarding information, disabling any email signatures, then signing the email with PGP/MIME.
Approve the USN mail on https://lists.ubuntu.com/mailman/admindb/ubuntu-security-announce. Ensure to reject duplicate mails from you (some list subscribers bounce mails back unmodified). Verify it went through in https://lists.ubuntu.com/archives/ubuntu-security-announce/.
- Update your local USN database and update the CVE tracker:
cd $UCT && bzr up && ./scripts/fetch-db database.pickle.bz2 && ./scripts/sync-from-usns.py
Run ./scripts/sync-from-usns.py -u if visual inspection of updates looks correct.
(This step is normally automated now.) To help reduce the load on security.archive.com, new packages in -security should be automatically copied to -updates at :28 and :58. For large updates (OOo, firefox, kernel, kdebase) be sure to verify that they did indeed get copied. If not, ping an archive admin about doing a pocket-copy from -security to -updates. Please note that security fake syncs and any packages with a new source orig.tar.gz will need to be manually copied to -updates by an archive admin.
Check for any outstanding LP bugs tied to the CVEs that are resolved with the USN. https://bugs.launchpad.net/bugs/cve/YYYY-NNNN
- Delete SRCPKG from Security PPA.
Sometimes a security vulnerability is covered by the media. In an effort to provide accurate information for users, PR and journalists, use the following process if your update has media coverage or is otherwise high profile.
- First and foremost, ensure someone is assigned to perform the update and is working on it
- In parallel to the update preparation, have someone else prepare the knowledgebase article
Create a KnowledgeBase article by navigating to MediaCoverageTemplate and saving it as https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/<Something>
- Fill in the details of the knowledgebase article using the template
- Get signoff from at least one other member of the security team
Update https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase#Media_coverage to include a link to your new article (TODO: automate?)
- Update the CVE in UCT to reference the knowledgebase article
- If there is a bug reference for this issue, update the bug to reference the knowledgebase article
- Update the published USN to reference the knowledgebase article in the 'Reference' section (see below)
- Optionally update the /topic in #ubuntu-hardened to include the knowledgebase article
Republishing/Editing a Published USN
Update the USN database with ssh -t people.canonical.com ~ubuntu-security/bin/edit-usn NNN-S, where NNN-S corresponds to the edited USN, eg 582-2.
Copy the updated USN database by running: ssh people.canonical.com "~ubuntu-security/bin/push-usn-db"
Create a new USN page for this by navigating to http://www.ubuntu.com/usn/update/usn-###-#. Via the command line: w3m -dump http://www.ubuntu.com/usn/update/usn-$USN. This should return an 'Ok' message (eg, 'Loading USN-1110-1: Ok'). If this returns 'No such USN: usn-xxx', wait a couple of minutes and try again.
Removing a Published USN
On people.canonical.com, create/verify a new database with the USN removed, and copy it into place:
$ cd ~ubuntu-security/usn # make a backup of the original $ cp ./database-all.pickle ./database-all.pickle.$(date +%Y-%m-%d) # create the new db: $ cd ~ubuntu-security/usn $ ../bzr-pulls/usn-tool/usn.py --export --db=./database-all.pickle > ./orig.yaml $ cp ./orig.yaml updated.yaml ... edit updated.yaml to remove the USN ... $ cat ./updated.yaml | ../bzr-pulls/usn-tool/usn.py --import --db=./database-all.pickle.new # verify the db $ ../bzr-pulls/usn-tool/usn.py --export --db=./database-all.pickle.new > ./verify.yaml $ diff ./updated.yaml ./verify.yaml # should be identical $ diff ./orig.yaml ./verify.yaml # should have only the USN removed # copy into place $ mv ./database-all.pickle.new ./database-all.pickle
- ask #webops to remove the USN from the usn microsite
Make sure ~/.ubuntu-cve-tracker.conf is fully configured (see the u-c-t README), and set the path to the ubuntu-cve-tracker bzr tree check-out: UCT=/path/to/ubuntu-cve-tracker
Grab the database.pickle file: cd $UCT; ./scripts/fetch-db database.pickle.bz2
set UQT to the ubuntu-qa-tools tree check-out: UQT=/path/to/ubuntu-qa-tools
Set up ~/.dput.cf with the appropriate Security PPA upload entries:
Now you can upload with dput. For example:
$ dput -s security:lucid ./*_source.changes # dry run $ dput security:lucid ./*_source.changes
Security Proposed PPA
This ppa is a public PPA which builds packages with only -security enabled. It is used to build packages that need wider testing. Packages can either be tested directly from this PPA, or more often pocket copied into the -proposed. A member of ubuntu-archive is required to copy packages from this PPA to -proposed.
To use the security-proposed PPA:
Set up ~/.dput.cf to have an entry like this for each release:
- Then can upload with: