UpdatePublication

Differences between revisions 141 and 202 (spanning 61 versions)
Revision 141 as of 2016-05-05 14:10:50
Size: 20720
Editor: jdstrand
Comment:
Revision 202 as of 2021-10-08 15:46:29
Size: 27590
Editor: mdeslaur
Comment:
Deletions are marked like this. Additions are marked like this.
Line 4: Line 4:
= Publishing an Update to Soyuz = = Publishing an Update to Launchpad =
Line 9: Line 9:
 0. Prepare your [[https://wiki.ubuntu.com/SecurityTeam/UpdatePublication#Local_ConfigurationLocal|local configuration]]
 0. Upload the updated source packages via dput to each release's Security PPA target. Notification about failed builds should be automatically sent to `security@ubuntu.com`.
 0. '''Note for cloud images, LXD images, Ubuntu on Windows images, cloud archive, Ubuntu Core and/or Ubuntu Touch''':
 0. Prepare your [[https://wiki.ubuntu.com/SecurityTeam/UpdatePublication#Local_Configuration|local configuration]]
 0. Upload the updated source packages via dput to each release's Security PPA target or for ESM, the ESM staging PPA). Notification about failed builds should be automatically sent to `security@ubuntu.com`.
 0. '''Note for cloud images, LXD images, Ubuntu on Windows images, cloud archive, and/or Ubuntu Core''':
Line 14: Line 14:
  * Ubuntu Core stable images/OS and kernel snaps are released on a cadence and therefore will bundle security updates every few weeks using a manual process (currently) that is managed by the Ubuntu Core team. Out-of-cadence emergency updates may be performed for urgent issues. If the update requires urgently updating Ubuntu Core, alert `JamieBennett`, `mvo`, `bueno` and `olli` that an update is coming that requires an emergency update.
  * Ubuntu Touch images are updated via OTA using a six week cadence. Security updates automatically flow from the archive into rc-proposed daily if the updated package doesn't exist in the [[https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/stable-phone-overlay/+packages|stable-phone-overlay ppa]]. If it does exist in the stable-phone-overlay, the package in the stable-phone-overlay PPA will need to be updated via the normal silo process. In this manner, security updates are bundled together every 6 weeks. Out-of-cadence emergency updates may be performed for urgent issues. If the update requires urgently updating the Ubuntu Touch stable images, alert `pmcgowan` and `sil2100` that an update is coming that requires an emergency update.
  * Ubuntu Core stable images/OS and kernel snaps are [[http://forum.snapcraft.io/t/core-snap-delivery-process/314/1|QA'd]] (snapd SRU's have similar [[http://forum.snapcraft.io/t/sru-verification-process/363/1|QA]]) and released on a [[https://lists.canonical.com/mailman/private/canonical-snapcraft/2016-November/003996.html|two-week cadence]] and therefore will bundle security updates every two weeks using a manual process (currently) that is managed by the Ubuntu Core team. Out-of-cadence emergency updates may be performed for urgent issues. If the update requires urgently updating Ubuntu Core, alert `mvo` (cc'ing `cachio`) from the snapd team for the core snap and `sil2100` (cc'ing `mclemenceau`) for the coreNN snaps that an update is coming that requires an emergency update.
Line 18: Line 17:
  * '''NOTE:''' if there is a problem with a buildd, ask `webops` in `#webops`. Eg: "webops: can you please kill the build for kde4libs 4:4.9.0a-0ubuntu2 on lamiak?". Can see the build farm at https://launchpad.net/builders/.   * '''NOTE:''' if there is a problem with a buildd, ask `webops` in `#is`. Eg: "webops: can you please kill the build for kde4libs 4:4.9.0a-0ubuntu2 on lamiak?". Can see the build farm at https://launchpad.net/builders/.
  * '''NOTE:''' For ESM: $UCT/scripts/sis-changes --action check-build $SRCPKG --ppa ubuntu-security/esm --include-eol
  * '''NOTE:''' If you want to check in all ESM and normal Ubuntu (proposed in this example): $UCT/scripts/sis-changes --action check-build $SRCPKG --esm-merge-ppa=ubuntu-security/esm --esm-merge-ppa=ubuntu-esm/esm-infra-security-staging --ppa=ubuntu-security-proposed/ppa --include-eol
Line 21: Line 22:
 0. If this is a sponsored update (for universe), edit the CVE entries in UCT to reflect the update.   * '''NOTE:''' For ESM: skip this step. Instead, use Launchpad to copy the package to 'Extended Security Maintenance [~ubuntu-esm/ubuntu/esm]'. You need to copy the binaries.'Copy existing binaries' radio must be selected.
  * Alternatively, instead of using the launchpad web interface, use the copy-package tool from the [[https://code.launchpad.net/~ubuntu-archive/ubuntu-archive-tools/trunk|lp:ubuntu-archive-tools tree]], like so:
   * [precise/esm] {{{${UAT}/copy-package --include-binaries --from ~ubuntu-security/ubuntu/esm --from-suite precise --to ~ubuntu-esm/ubuntu/esm --to-suite precise $SRCPKG}}}
   * [trusty/esm] {{{${UAT}/copy-package --include-binaries --from ~ubuntu-esm/ubuntu/esm-infra-security-staging --from-suite trusty --to ~ubuntu-esm/ubuntu/esm-infra-security --to-suite trusty $SRCPKG}}}
 0. If this is a sponsored update (for universe), edit the CVE entries in UCT to reflect the update. Update the [[https://wiki.ubuntu.com/SecurityTeam/Meeting|Security Team Meeting]] page to credit the person who provided the debdiff.
 0. If this is an embargoed update, mv the CVE entries from `embargoed/` to
 `$UCT/active/`. `bzr ci -m "unembargo $SRCPKG"` in embargoed, `bzr add ; bzr ci -m "unembargo $SRCPKG"` in UCT.
Line 32: Line 39:
 In theory, the copy-package can also be used by an archive admin, followed by any override adjustments (you can use the ```find-bin-overrides``` script from lp:ubuntu-qa-tools/security-tools for a list of overrides). However, this tool may not work with the private ubuntu-security PPA. Eg, to unembargo sudo 1.7.2p7-1ubuntu2 on utopic, you would ask the archive admin to run the following (see ArchiveAdministration for more information):{{{  In theory, the copy-package can also be used by an archive admin, followed by any override adjustments (you can use the ```find-bin-overrides``` script from [[https://code.launchpad.net/ubuntu-qa-tools | lp:ubuntu-qa-tools/security-tools]] for a list of overrides). However, this tool may not work with the private ubuntu-security PPA. Eg, to unembargo sudo 1.7.2p7-1ubuntu2 on utopic, you would ask the archive admin to run the following (see ArchiveAdministration for more information):{{{
Line 44: Line 51:
Please see [[SecurityTeam/UpdatePublication/Kernel]] for information on the kernel update process and cycle.
Line 55: Line 64:
 0. If applicable, verify that the Certified Public Cloud, Cloud Archive, Ubuntu Core and/or Ubuntu Touch teams have acknowledged the urgent update notification (see '[[https://wiki.ubuntu.com/SecurityTeam/UpdatePublication#Upload.2BAC8-Build.2BAC8-Publish|Note for cloud images, cloud archive, Ubuntu Core and/or Ubuntu Touch]]')  0. If applicable, verify that the Certified Public Cloud, Cloud Archive, and/or Ubuntu Core teams have acknowledged the urgent update notification (see '[[https://wiki.ubuntu.com/SecurityTeam/UpdatePublication#Upload.2BAC8-Build.2BAC8-Publish|Note for cloud images, cloud archive and/or Ubuntu Core]]')
Line 58: Line 67:
  0. For an old issue that needs correction or continuation, start with the issue's original USN, keep NNN and increase S. (e.g. original issue was 42-1, updated USN will be 42-'''2''').
  0. For a new issue that affects different software with identical CVEs, get a new USN normally for the first source package, and then keep NNN and increase S for each additional source package. (e.g. CVE-2008-1693 affected both poppler and koffice, so 603-1 was used for poppler and 603-2 was used for koffice). Please note that different versions of the same software (e.g. emacs21 and emacs22) should not do this, but instead use a single USN with S=1.
 0. To create the USN template script, run: `$UCT/scripts/sis-changes --download /tmp/pending $SRCPKG && cd /tmp/pending && $UCT/scripts/sis-generate-usn $USN *.changes > ~/new-usn.sh`
  0. For an old issue that needs correction or continuation, start with the issue's original USN, keep NNN and increase S. (e.g. original issue was 42-1, updated USN will be 42-'''2'''). Adjust the following command for your USN number: `USN=42-2`
  0. For a new issue that affects different software with identical CVEs, get a new USN normally for the first source package, and then keep NNN and increase S for each additional source package. (e.g. CVE-2008-1693 affected both poppler and koffice, so 603-1 was used for poppler and 603-2 was used for koffice). Please note that different versions of the same software (e.g. emacs21 and emacs22) should not do this, but instead use a single USN with S=1. The only exception to this last rule is when you are publishing different versions of the same software but at different points in time. In this case you will just increase S for each of them.
  0. For an ESM issue, if the update has already been published for a current release, use the NNN of the published USN for release updates and increment S (e.g. original issue was 3282-1, ESM USN will be 3282-2).
 0.
To create the USN template script, run:
  * For Ubuntu Archive only (from private PPA): {{{
$UCT/scripts/sis-changes --download /tmp/pending-$USN $SRCPKG && cd /tmp/pending-$USN && $UCT/scripts/sis-generate-usn $USN *.changes > ~/new-usn-$USN.sh
}}}
     * From the proposed PPA: {{{
$UCT/scripts/sis-changes --download /tmp/pending-$USN $SRCPKG --ppa ubuntu-security-proposed/ppa && cd /tmp/pending-$USN && $UCT/scripts/sis-generate-usn $USN *.changes > ~/new-usn-$USN.sh
}}}
  * For Trusty ESM and ESM infra security staging: {{{
$UCT/scripts/sis-changes --download /tmp/pending-$USN $SRCPKG --ppa ubuntu-esm/esm-infra-security-staging --include-eol && cd /tmp/pending-$USN && $UCT/scripts/sis-generate-usn --no-new-warn $USN *.changes > ~/new-usn-$USN.sh
}}}
  * For ESM and normal releases: {{{
$UCT/scripts/sis-changes --download /tmp/pending-$USN $SRCPKG --esm-merge-ppa=ubuntu-esm/esm-infra-security-staging --ppa ubuntu-security-proposed/ppa --include-eol && cd /tmp/pending-$USN && $UCT/scripts/sis-generate-usn --no-new-warn $USN *.changes > ~/new-usn-$USN.sh
}}}
Line 64: Line 86:
 0. Edit `~/new-usn.sh` to include a correct title, summary, action, description, and then limit the binary list to only those affected by the USN (`umt compare-bin` may be helpful in determing the affected binaries). Leave all URLs as-is. Have the description proofread by somebody else.   * When uploading to a PPA, `sis-changes` may report that `INFO: new binary '<pkg>.deb' in release '<release>', is packages_mirror up to date?`, which is safe to ignore.
  * If `sis-changes` fails with `error: USN NNN-S not in the database`, run `cd $UCT && ./scripts/fetch-db database.pickle.bz2`. Then, re-run `sis-generate-usn`: `$UCT/scripts/sis-generate-usn <...> $USN *.changes > ~/new-usn-$USN.sh`.
 0. Edit `~/new-usn.sh` to include a correct title, summary, action, description, and then limit the binary list to only those affected by the USN . Leave all URLs as-is. Have the description proofread by somebody else.
Line 68: Line 92:
  * Set the title to the package name and reflect how upstream refers to it. If it affects multiple source packages with the same upstream, the title should only include the one, general upstream name for the package.
  * The summary should follow ubuntu source package naming conventions. If it affects multiple source packages, list the packages.
  * There is [[https://git.launchpad.net/ubuntu-cve-tracker/tree/README.usn |a guide on writing the description]] in UCT. Updates to the devel release are not included in USNs.
  * Fill in the issue-summary using the upstream name. If there were multiple different CVEs, use the `Several security issues were fixed in <pkg>.`. Otherwise, choose one of the templates or, if it doesn’t match any, write your own but keep it high-level, non-technical, and without deviating too much from the templates.
  * Sometimes the "source-description" field in the generated USN is unhelpful or inaccurate. In those situations, you'll need to update package_info_overrides.json with a new description that reflects how the package maintainers describe the package but less verbosely by running `$UCT/scripts/add_meta_info.py <pkg-src-name> <upstream-name> "<desc>"`.
  * Under `Reduce to minimum binaries`, you can generally remove the -doc and -udeb lines from the binary list. `umt compare-bin` may be helpful in determing the affected binaries. Remove the -header lines if there weren’t changes to .h files.
  * Limit the length of the description text to 75 characters.
Line 69: Line 100:
 0. Wait until the packages are actually mirrored to the archive; the publisher runs hourly at :03, and updates should usually appear on `security.ubuntu.com` within 20-40 minutes, depending on the size and number of binary packages (note 0403 UTC publication run is skipped due to the Contents generation job). If it is more than 3 hours before the packages are mirrored, then ask #is. To verify that the packages have arrived, run: `ssh people.canonical.com "~ubuntu-security/bin/check-upload $USN"`  0. Wait until the packages are actually mirrored to the archive; the publisher runs hourly at :03, and updates should usually appear on `security.ubuntu.com` within 20-40 minutes, depending on the size and number of binary packages (note 0403 UTC publication run is skipped due to the Contents generation job). If it is more than 3 hours before the packages are mirrored, then ask ~is. To verify that the packages have arrived, run: `ssh people.canonical.com "~ubuntu-security/bin/check-upload $USN"`
Line 71: Line 102:
 0. Create a new USN page for this by running: `w3m -dump http://www.ubuntu.com/usn/update/usn-$USN; for i in $(seq 1 4); do w3m -dump http://www.ubuntu.com/usn/usn-$USN | awk '/Ubuntu Security/,/Back to top$/' ; done` This should return an 'Ok' message (eg, 'Loading USN-1110-1: Ok') followed by a text dump of the page. If this returns 'No such USN: usn-xxx', wait a couple of minutes and try again. Bugs in the microsite can be filed at [[https://bugs.launchpad.net/usn-website|https://bugs.launchpad.net/usn-website]]. Urgent issues for problems that are identified as problems with the USN microsite should be directed to mars (coding errors) or #webops (all other issues).   * If this returns `/home/ubuntu-security/public_html/usn/database-all.json.new' already exists`, it’s possible someone else is also pushing a USN, wait a bit before trying again.
 0. Update your local USN database, publish the new USN to the USN website, and track the deployment status by visiting the link printed by the script:
  0. `cd $UCT && ./scripts/fetch-db database.pickle.bz2 && ./scripts/publish-usn-to-website -a $USN`
     * If it fails with `ERROR: Ensure that <USN> is in [...]/database.pickle and try again`, it may be possible that someone else is also publishing simultaneously, ask around and if that is the case wait for them to finish, then retry.
     * If it fails with
       {{{
macaroonbakery.httpbakery._error.InteractionError: cannot start interactive session: cannot get <url>
ERROR: Failed to publish <USN> pickle to website API
}}}
       if the previous fetch-db step completed successfully, (you see the `Download complete. File saved to 'database.pickle'` line), just try rerunning the publish-usn-to-website script.
     * If it fails with `504 Gateway Time-out or 408 Request Time-out`, make sure your wifi and VPN has IPv6 disabled, close and reopen your browser, and re-run the publish-usn-to-website script. Failing that, keep re-trying every once in a while for an hour or two.
  0. Navigate to the linked USN page.

  '''Important''': The script must succeed or the new USN will not be displayed on the website. If the deployment job continues to fail after retrying, you'll need to escalate in order to get a fix. Join `#web-team`, start a message with `@webteam` and describe the problem. If you don't get a response, join `#is` and ping the vanguard listed in the channel topic and be make sure to mention that you don't have any deep knowledge of the publishing architecture and point them at the usn.ubuntu.com architecture document.

  Proceed on with sending the USN email but you '''must''' follow up and ensure that the deployment job eventually succeeds.
Line 78: Line 124:
 This can be done any number of ways. Typically it involves forwarding the email inline, adjusting the From to be from you, adjusting the Subject to remove any Fwd text, removing any attachments, adjusting the body text to remove forwarding information, disabling any email signatures, then signing the email with PGP/MIME.  This can be done any number of ways. Typically it involves forwarding the email inline, adjusting the From to be from you, adjusting the Subject to remove any Fwd text, removing any attachments, adjusting the body text to remove forwarding information, disabling any email signatures, then signing the email with PGP/MIME. If you need help setting up headers on Thunderbird, you can follow this guide: https://wiki.mozilla.org/Thunderbird:Help_Documentation:Mail-Followup-To_and_Mail-Reply-To
Line 80: Line 127:
 0. Update your local USN database and update the CVE tracker:
  0. `cd $UCT && bzr up && ./scripts/fetch-db database.pickle.bz2 && ./scripts/sync-from-usns.py`
 0. Update the CVE tracker:
  0. `cd $UCT && git pull --ff-only && ./scripts/sync-from-usns.py`
Line 85: Line 132:
   * Run `bzr status ; bzr diff` to see what will be checked-in. Run `bzr ci -m $USN` to check in the changes.   0. `./scripts/cve_need_retire`
   * Run `./scripts/cve_need_retire -u` if visual inspection of updates looks correct.
  0. Run `git status ; git diff` to see what will be checked-in. Check in the changes and push the repo.
Line 89: Line 138:
 0. Delete SRCPKG from Security PPA.  0. Delete SRCPKG from Security PPA. (WARNING! Failure to perform this step may result in tarring and feathering!)
Line 107: Line 156:
 0. Update the USN database with `ssh -t people.canonical.com ~ubuntu-security/bin/edit-usn NNN-S`, where NNN-S corresponds to the edited USN, eg 582-2.
 0. Copy the updated USN database by running: `ssh people.canonical.com "~ubuntu-security/bin/push-usn-db"`
 0. Create a new USN page for this by navigating to http://www.ubuntu.com/usn/update/usn-###-#. Via the command line: `w3m -dump http://www.ubuntu.com/usn/update/usn-$USN`. This should return an 'Ok' message (eg, 'Loading USN-1110-1: Ok'). If this returns 'No such USN: usn-xxx', wait a couple of minutes and try again.
 0. Decide which USN you'll be updating, eg 582-2:
 `USN=NNN-S`
 0. Update the USN database:
 `ssh -t people.canonical.com ~ubuntu-security/bin/edit-usn $USN`
 0. Copy the updated USN database by running:
 `ssh people.canonical.com "~ubuntu-security/bin/push-usn-db"`
 0. Update your local USN database, publish the new USN to the USN website, and track the deployment status by visiting the link printed by the script:
 `cd $UCT && ./scripts/fetch-db database-all.pickle.bz2 && ./scripts/publish-usn-to-website -u -d ./database-all.pickle $USN`
Line 112: Line 166:
The following commands should be okay to run on lillypilly. However, the export to yaml does take almost 5GB of ram and can take several minutes. It may be preferred to copy the unmodified and modified versions of the database locally to generate the before and after yaml files for verification. The actual deletion takes much less time/ram.
Line 114: Line 170:
# make a backup of the original
$ cp ./database-all.pickle ./database-all.pickle.$(date +%Y-%m-%d)
# create the new db:
# create an unmodified yaml file for validation:
Line 118: Line 172:
$ ../bzr-pulls/usn-tool/usn.py --export --db=./database-all.pickle > ./orig.yaml
$ cp ./orig.yaml updated.yaml
... edit updated.yaml to remove the USN ...
$ cat ./updated.yaml | ../bzr-pulls/usn-tool/usn.py --import --db=./database-all.pickle.new
$ ../git-pulls/usn-tool/usn.py --export --db=./database-all.pickle > ./orig.yaml
# (on lillypilly, that needs to be ../bin/usn.sh --export --db=./database-all.pickle > ./orig.yaml)
# delete the usn from the db:
$ ../git-pulls/usn-tool/delete_usn.py --db=./database-all.pickle $USN
Line 123: Line 177:
$ ../bzr-pulls/usn-tool/usn.py --export --db=./database-all.pickle.new > ./verify.yaml
$ diff ./updated.yaml ./verify.yaml # should be identical
$ ../git-pulls/usn-tool/usn.py --export --db=./database-all.pickle > ./verify.yaml
# (on lillypilly, that needs to be ../bin/usn.sh --export --db=./database-all.pickle > ./verify.yaml)
Line 126: Line 180:
# copy into place
$ mv ./database-all.pickle.new ./database-all.pickle
# if something went wrong with the deletion, the delete_usn.py script saves a copy
# of the database before modifying in database-all.pickle-YYYY-MM-DD-HH:MM:SS
Line 130: Line 184:
 0. ask #webops to remove the USN from the usn microsite  0. Update your local USN database, remove the USN from the USN website, and track the deployment status by visiting the link printed by the script:
 `cd $UCT && ./scripts/fetch-db database-all.pickle.bz2 && ./scripts/publish-usn-to-website -r -d ./database-all.pickle $USN`
Line 134: Line 189:
 0. Make sure {{{~/.ubuntu-cve-tracker.conf}}} is fully configured (see the u-c-t README), and set the path to the ubuntu-cve-tracker bzr tree check-out: {{{UCT=/path/to/ubuntu-cve-tracker}}}  0. Make sure {{{~/.ubuntu-cve-tracker.conf}}} is fully configured (see the u-c-t README), and set the path to the ubuntu-cve-tracker git tree check-out: {{{UCT=/path/to/ubuntu-cve-tracker}}}
Line 169: Line 224:
$ umt upload -d security-proposed:xenial # if using the umt tool

While handling a security update, one must prepare the upload, then follow these steps to publish.

Publishing an Update to Launchpad

Upload/Build/Publish

Stable releases

  1. If also unembargoing a development release from the Ubuntu Security PPA, unembargo the development release first (see below).

  2. Prepare your local configuration

  3. Upload the updated source packages via dput to each release's Security PPA target or for ESM, the ESM staging PPA). Notification about failed builds should be automatically sent to security@ubuntu.com.

  4. Note for cloud images, LXD images, Ubuntu on Windows images, cloud archive, and/or Ubuntu Core:

    • If the update requires urgently updating the cloud images, LXD images, or Ubuntu on Windows images, email cpc@canonical.com to notify the Certified Public Cloud team that an update is coming which requires regenerating the images (specifics are not required so as not to accidentally break embargo) and ask for an acknowledgement of the email. If the email is not acknowledged in a timely fashion, escalate to rcj, Odd_Bloke, and gaughen in #server on Canonical irc. Package manifests can be found in http://cloud-images.ubuntu.com/releases/<release>/release/*.manifest

    • If the update requires urgently updating the cloud archive, alert jamespage that an update is coming that requires a copy from the security pocket to the cloud archive. A list of packages in each cloud archive release can be found at http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/ (Actual repository is located here: http://ubuntu-cloud.archive.canonical.com/ )

    • Ubuntu Core stable images/OS and kernel snaps are QA'd (snapd SRU's have similar QA) and released on a two-week cadence and therefore will bundle security updates every two weeks using a manual process (currently) that is managed by the Ubuntu Core team. Out-of-cadence emergency updates may be performed for urgent issues. If the update requires urgently updating Ubuntu Core, alert mvo (cc'ing cachio) from the snapd team for the core snap and sil2100 (cc'ing mclemenceau) for the coreNN snaps that an update is coming that requires an emergency update.

  5. Set the name of the source package being updated: export SRCPKG="srcpkg1 srcpkg2..."

  6. Wait for the finished builds on all supported architectures to finish and appear at the Ubuntu Security PPA: $UCT/scripts/sis-changes --action check-build $SRCPKG

    • NOTE: if there is a problem with a buildd, ask webops in #is. Eg: "webops: can you please kill the build for kde4libs 4:4.9.0a-0ubuntu2 on lamiak?". Can see the build farm at https://launchpad.net/builders/.

    • NOTE: For ESM: $UCT/scripts/sis-changes --action check-build $SRCPKG --ppa ubuntu-security/esm --include-eol

    • NOTE: If you want to check in all ESM and normal Ubuntu (proposed in this example): $UCT/scripts/sis-changes --action check-build $SRCPKG --esm-merge-ppa=ubuntu-security/esm --esm-merge-ppa=ubuntu-esm/esm-infra-security-staging --ppa=ubuntu-security-proposed/ppa --include-eol

  7. Unembargo by copying the packages into the archive accepted queue with: $UQT/security-tools/unembargo $SRCPKG

    • NOTE: if you are using unembargo to publish from a non-private PPA (eg, ubuntu-security-proposed or ubuntu-mozilla-security), you may have to also get an archive admin to adjust overrides (use find-bin-overrides and give to an archive admin).

    • NOTE: For ESM: skip this step. Instead, use Launchpad to copy the package to 'Extended Security Maintenance [~ubuntu-esm/ubuntu/esm]'. You need to copy the binaries.'Copy existing binaries' radio must be selected.

    • Alternatively, instead of using the launchpad web interface, use the copy-package tool from the lp:ubuntu-archive-tools tree, like so:

      • [precise/esm] ${UAT}/copy-package --include-binaries --from ~ubuntu-security/ubuntu/esm --from-suite precise --to ~ubuntu-esm/ubuntu/esm --to-suite precise $SRCPKG

      • [trusty/esm] ${UAT}/copy-package --include-binaries --from ~ubuntu-esm/ubuntu/esm-infra-security-staging --from-suite trusty --to ~ubuntu-esm/ubuntu/esm-infra-security --to-suite trusty $SRCPKG

  8. If this is a sponsored update (for universe), edit the CVE entries in UCT to reflect the update. Update the Security Team Meeting page to credit the person who provided the debdiff.

  9. If this is an embargoed update, mv the CVE entries from embargoed/ to $UCT/active/. bzr ci -m "unembargo $SRCPKG" in embargoed, bzr add ; bzr ci -m "unembargo $SRCPKG" in UCT.

Development release

In general, security updates are not under embargo and can be uploaded directly to the development release without a USN. For updates that are embargoed in the development release, the process is similar to the above, but different due to LP: #334858.

  1. Prepare your local configuration

  2. Set the name of the source package being updated: export SRCPKG="srcpkg1 srcpkg2..."

  3. Upload the updated source packages via dput to the development release's Security PPA target. The distribution name should not contain the '-security' suffix. Notification about failed builds should be automatically sent to security@ubuntu.com.

  4. Wait for the finished builds on all supported architectures to finish and appear at the Ubuntu Security PPA: $UCT/scripts/sis-changes --include-devel -r <development release> --action check-build $SRCPKG. NOTE: due to a bug in sis-changes, this doesn't actually work so you'll have to check the PPA manually.

  5. When the issue is public, ask an archive admin to unembargo the package using $UQT/security-tools/unembargo $SRCPKG. Eg, to unembargo curl on utopic, ask an archive admin to use:

    $ $UQT/security-tools/unembargo --ppa=ubuntu-security -r utopic --pocket=proposed curl

    In theory, the copy-package can also be used by an archive admin, followed by any override adjustments (you can use the find-bin-overrides script from lp:ubuntu-qa-tools/security-tools for a list of overrides). However, this tool may not work with the private ubuntu-security PPA. Eg, to unembargo sudo 1.7.2p7-1ubuntu2 on utopic, you would ask the archive admin to run the following (see ArchiveAdministration for more information):

    $ copy-package -b --ppa=ubuntu-security -s utopic --to-primary --to-suite utopic-proposed -e 1.7.2p7-1ubuntu2 sudo
    $ change-override -c universe -s utopic-proposed sudo-ldap
    • This may need to be done by a someone who is both a member of ubuntu-security and ubuntu-archive

  6. Delete SRCPKG from Security PPA for the development release. This should be done before unembargoing the stable releases to ensure it is copied only to the release pocket, and not the security pocket for the development release.

  7. At this point, the publication for the development release is completed (a USN is not issued for the development release). Proceed to unembargo the stable releases as needed (see above).

Partner

Occasionally (usually only in the case of an emergency) the Ubuntu Security team may be asked to update or sponsor a package in the Canonical partner archive. After verifying that the source packaging is correct for partner, uploading is a simple matter of dputting the package to the Ubuntu archive (the Section specified in the packaging tells Launchpad to put it in partner). Launchpad restricts copies from PPAs to the partner archive, so the packages must be uploaded directly to the archive (requires membership in canonical-partner-dev).

Kernel publication

Please see SecurityTeam/UpdatePublication/Kernel for information on the kernel update process and cycle.

  1. Prepare your local configuration

  2. Set the name of the source package being updated, using the ABI Packages page as a guide (git branches with stars are not handled by the Security Team): export SRCPKG="srcpkg1 srcpkg2..."

  3. Make sure to check the ABI is consistent: $UQT/security-tools/kernel-abi-check

    • This will say if the ABI has changed, and what the version was in the last published USN
    • Make sure you have the latest USN database: cd $UCT; ./scripts/fetch-db database.pickle.bz2

  4. Make sure all the packages are in the archive: $UCT/scripts/sis-changes --action check-build $SRCPKG --ppa ubuntu --pocket Security -r REL

Announce Publication

(for main/restricted publications in stable releases)

  1. If applicable, verify that the Certified Public Cloud, Cloud Archive, and/or Ubuntu Core teams have acknowledged the urgent update notification (see 'Note for cloud images, cloud archive and/or Ubuntu Core')

  2. Assign a USN (format is NNN-S, and the following instructions assume $USN has been set as desired):
    1. For a new issue, run: USN=$(ssh people.canonical.com "~ubuntu-security/bin/get-next-usn" $SRCPKG) optionally including any extra information explaining why the USN was issued.

    2. For an old issue that needs correction or continuation, start with the issue's original USN, keep NNN and increase S. (e.g. original issue was 42-1, updated USN will be 42-2). Adjust the following command for your USN number: USN=42-2

    3. For a new issue that affects different software with identical CVEs, get a new USN normally for the first source package, and then keep NNN and increase S for each additional source package. (e.g. CVE-2008-1693 affected both poppler and koffice, so 603-1 was used for poppler and 603-2 was used for koffice). Please note that different versions of the same software (e.g. emacs21 and emacs22) should not do this, but instead use a single USN with S=1. The only exception to this last rule is when you are publishing different versions of the same software but at different points in time. In this case you will just increase S for each of them.
    4. For an ESM issue, if the update has already been published for a current release, use the NNN of the published USN for release updates and increment S (e.g. original issue was 3282-1, ESM USN will be 3282-2).
  3. To create the USN template script, run:
    • For Ubuntu Archive only (from private PPA):

      $UCT/scripts/sis-changes --download /tmp/pending-$USN $SRCPKG && cd /tmp/pending-$USN && $UCT/scripts/sis-generate-usn $USN *.changes > ~/new-usn-$USN.sh
      • From the proposed PPA:

        $UCT/scripts/sis-changes --download /tmp/pending-$USN $SRCPKG --ppa ubuntu-security-proposed/ppa && cd /tmp/pending-$USN && $UCT/scripts/sis-generate-usn $USN *.changes > ~/new-usn-$USN.sh
    • For Trusty ESM and ESM infra security staging:

      $UCT/scripts/sis-changes --download /tmp/pending-$USN $SRCPKG --ppa ubuntu-esm/esm-infra-security-staging --include-eol && cd /tmp/pending-$USN && $UCT/scripts/sis-generate-usn --no-new-warn $USN *.changes > ~/new-usn-$USN.sh
    • For ESM and normal releases:

      $UCT/scripts/sis-changes --download /tmp/pending-$USN $SRCPKG --esm-merge-ppa=ubuntu-esm/esm-infra-security-staging --ppa ubuntu-security-proposed/ppa --include-eol && cd /tmp/pending-$USN && $UCT/scripts/sis-generate-usn --no-new-warn $USN *.changes > ~/new-usn-$USN.sh
    • For announcing kernels, add --kernel-mode --filter-bins '^linux-image-\d' --no-new-warn to the sis-generate-usn command line to make binary package lists saner. --no-new-warn is only necessary when publishing a new source tarball

    • For announcing kernels that have been already released, add --kernel-mode --filter-bins '^linux-image-\d' --no-new-warn --ppa ubuntu --pocket Security -r REL[,REL] to the sis-changes command line. --no-new-warn is only necessary when publishing a new source tarball

    • For announcing Mozilla updates, see https://wiki.ubuntu.com/SecurityTeam/PublicationNotes#Mozilla%20Publishing

    • When uploading to a PPA, sis-changes may report that INFO: new binary '<pkg>.deb' in release '<release>', is packages_mirror up to date?, which is safe to ignore.

    • If sis-changes fails with error: USN NNN-S not in the database, run cd $UCT && ./scripts/fetch-db database.pickle.bz2. Then, re-run sis-generate-usn: $UCT/scripts/sis-generate-usn <...> $USN *.changes > ~/new-usn-$USN.sh.

  4. Edit ~/new-usn.sh to include a correct title, summary, action, description, and then limit the binary list to only those affected by the USN . Leave all URLs as-is. Have the description proofread by somebody else.

    • When announcing kernels, look at the *source.changes file that is now in /tmp/pending to make sure the CVEs for all the releases since the last USN are present. Make sure no CVE fixes have been reverted and are listed in ~/new-usn.sh by mistake, and that all the changelog entries since the last update are listed.

    • If the update is a regression fix or does not fix CVEs, be sure to create a tracking bug and have new-usn.sh reference at least this one bug (eg, something like 'usn.py $DB $USN --cve "https://launchpad.net/bugs/NNNNNNN"'

    • If the update is a regression fix, be sure to replace "vulnerability" with "regression" in the title
    • Set the title to the package name and reflect how upstream refers to it. If it affects multiple source packages with the same upstream, the title should only include the one, general upstream name for the package.
    • The summary should follow ubuntu source package naming conventions. If it affects multiple source packages, list the packages.
    • There is a guide on writing the description in UCT. Updates to the devel release are not included in USNs.

    • Fill in the issue-summary using the upstream name. If there were multiple different CVEs, use the Several security issues were fixed in <pkg>.. Otherwise, choose one of the templates or, if it doesn’t match any, write your own but keep it high-level, non-technical, and without deviating too much from the templates.

    • Sometimes the "source-description" field in the generated USN is unhelpful or inaccurate. In those situations, you'll need to update package_info_overrides.json with a new description that reflects how the package maintainers describe the package but less verbosely by running $UCT/scripts/add_meta_info.py <pkg-src-name> <upstream-name> "<desc>".

    • Under Reduce to minimum binaries, you can generally remove the -doc and -udeb lines from the binary list. umt compare-bin may be helpful in determing the affected binaries. Remove the -header lines if there weren’t changes to .h files.

    • Limit the length of the description text to 75 characters.
  5. To populate the USN database with the new USN details and generate the template email (sent to security@ubuntu.com), run: bash ~/new-usn.sh

  6. Wait until the packages are actually mirrored to the archive; the publisher runs hourly at :03, and updates should usually appear on security.ubuntu.com within 20-40 minutes, depending on the size and number of binary packages (note 0403 UTC publication run is skipped due to the Contents generation job). If it is more than 3 hours before the packages are mirrored, then ask ~is. To verify that the packages have arrived, run: ssh people.canonical.com "~ubuntu-security/bin/check-upload $USN"

  7. Once packages are in the archive and you are satisfied with the email text (from new-ush.sh), copy the updated USN database by running: ssh people.canonical.com "~ubuntu-security/bin/push-usn-db"

    • If this returns /home/ubuntu-security/public_html/usn/database-all.json.new' already exists, it’s possible someone else is also pushing a USN, wait a bit before trying again.

  8. Update your local USN database, publish the new USN to the USN website, and track the deployment status by visiting the link printed by the script:
    1. cd $UCT && ./scripts/fetch-db database.pickle.bz2 && ./scripts/publish-usn-to-website -a $USN

      • If it fails with ERROR: Ensure that <USN> is in [...]/database.pickle and try again, it may be possible that someone else is also publishing simultaneously, ask around and if that is the case wait for them to finish, then retry.

      • If it fails with
        • macaroonbakery.httpbakery._error.InteractionError: cannot start interactive session: cannot get <url>
          ERROR: Failed to publish <USN> pickle to website API

          if the previous fetch-db step completed successfully, (you see the Download complete. File saved to 'database.pickle' line), just try rerunning the publish-usn-to-website script.

      • If it fails with 504 Gateway Time-out or 408 Request Time-out, make sure your wifi and VPN has IPv6 disabled, close and reopen your browser, and re-run the publish-usn-to-website script. Failing that, keep re-trying every once in a while for an hour or two.

    2. Navigate to the linked USN page.

      Important: The script must succeed or the new USN will not be displayed on the website. If the deployment job continues to fail after retrying, you'll need to escalate in order to get a fix. Join #web-team, start a message with @webteam and describe the problem. If you don't get a response, join #is and ping the vanguard listed in the channel topic and be make sure to mention that you don't have any deep knowledge of the publishing architecture and point them at the usn.ubuntu.com architecture document.

      Proceed on with sending the USN email but you must follow up and ensure that the deployment job eventually succeeds.

  9. Once the USN is available on the USN website, GPG sign and send the USN email with the following headers:
    To: ubuntu-security-announce@lists.ubuntu.com
    Reply-to: Ubuntu Security <security@ubuntu.com>
    Mail-Followup-to: Ubuntu Security <security@ubuntu.com>

    This can be done any number of ways. Typically it involves forwarding the email inline, adjusting the From to be from you, adjusting the Subject to remove any Fwd text, removing any attachments, adjusting the body text to remove forwarding information, disabling any email signatures, then signing the email with PGP/MIME. If you need help setting up headers on Thunderbird, you can follow this guide: https://wiki.mozilla.org/Thunderbird:Help_Documentation:Mail-Followup-To_and_Mail-Reply-To

  10. Approve the USN mail on https://lists.ubuntu.com/mailman/admindb/ubuntu-security-announce. Ensure to reject duplicate mails from you (some list subscribers bounce mails back unmodified). Verify it went through in https://lists.ubuntu.com/archives/ubuntu-security-announce/.

  11. Update the CVE tracker:
    1. cd $UCT && git pull --ff-only && ./scripts/sync-from-usns.py

      • For any missing CVEs, run ./scripts/active_edit -p PACKAGE -c CVE for each package and CVE that was fixed by the USN, and fill in the details in active/CVE with an editor.

      • Run ./scripts/sync-from-usns.py -u if visual inspection of updates looks correct.

      • If visual inspection indicates that unwanted USNs will be synced, run ./scripts/sync-from-usns.py -u --usn=$USN

    2. ./scripts/cve_need_retire

      • Run ./scripts/cve_need_retire -u if visual inspection of updates looks correct.

    3. Run git status ; git diff to see what will be checked-in. Check in the changes and push the repo.

  12. (This step is normally automated now.) To help reduce the load on security.archive.com, new packages in -security should be automatically copied to -updates at :28 and :58. For large updates (OOo, firefox, kernel, kdebase) be sure to verify that they did indeed get copied. If not, ping an archive admin about doing a pocket-copy from -security to -updates. Please note that security fake syncs and any packages with a new source orig.tar.gz will need to be manually copied to -updates by an archive admin.

    • IMPORTANT: the copier verifies that the new changelog in -security is a direct descendant of what is in -updates if the package currently exists in -updates. This helps ensure that a change in -updates is not reverted by a -security update. If this verification step fails, the package will not be automatically copied and a security team member must verify the changes are ok, and ask an archive admin to copy it manually.

  13. Check for any outstanding LP bugs tied to the CVEs that are resolved with the USN. https://bugs.launchpad.net/bugs/cve/YYYY-NNNN

  14. Delete SRCPKG from Security PPA. (WARNING! Failure to perform this step may result in tarring and feathering!)

Media coverage

Sometimes a security vulnerability is covered by the media. In an effort to provide accurate information for users, PR and journalists, use the following process if your update has media coverage or is otherwise high profile.

  1. First and foremost, ensure someone is assigned to perform the update and is working on it
  2. In parallel to the update preparation, have someone else prepare the knowledgebase article
  3. Create a KnowledgeBase article by navigating to MediaCoverageTemplate and saving it as https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/<Something>

  4. Fill in the details of the knowledgebase article using the template
  5. Get signoff from at least one other member of the security team
  6. Update https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase#Media_coverage to include a link to your new article (TODO: automate?)

  7. Send an email to 'pr@canonical.com' and 'support-team@lists.canonical.com', CC'ing 'security@ubuntu.com' and any other relevant individuals

  8. Update the CVE in UCT to reference the knowledgebase article
  9. If there is a bug reference for this issue, update the bug to reference the knowledgebase article
  10. Update the published USN to reference the knowledgebase article in the 'Reference' section (see below)
  11. Optionally update the /topic in #ubuntu-hardened to include the knowledgebase article

Republishing/Editing a Published USN

  1. Decide which USN you'll be updating, eg 582-2:

    USN=NNN-S

  2. Update the USN database:

    ssh -t people.canonical.com ~ubuntu-security/bin/edit-usn $USN

  3. Copy the updated USN database by running:

    ssh people.canonical.com "~ubuntu-security/bin/push-usn-db"

  4. Update your local USN database, publish the new USN to the USN website, and track the deployment status by visiting the link printed by the script:

    cd $UCT && ./scripts/fetch-db database-all.pickle.bz2 && ./scripts/publish-usn-to-website -u -d ./database-all.pickle $USN

Removing a Published USN

The following commands should be okay to run on lillypilly. However, the export to yaml does take almost 5GB of ram and can take several minutes. It may be preferred to copy the unmodified and modified versions of the database locally to generate the before and after yaml files for verification. The actual deletion takes much less time/ram.

  1. On people.canonical.com, create/verify a new database with the USN removed, and copy it into place:

    $ cd ~ubuntu-security/usn
    # create an unmodified yaml file for validation:
    $ cd ~ubuntu-security/usn
    $ ../git-pulls/usn-tool/usn.py --export --db=./database-all.pickle > ./orig.yaml
    # (on lillypilly, that needs to be ../bin/usn.sh --export --db=./database-all.pickle > ./orig.yaml)
    # delete the usn from the db:
    $ ../git-pulls/usn-tool/delete_usn.py --db=./database-all.pickle $USN
    # verify the db
    $ ../git-pulls/usn-tool/usn.py --export --db=./database-all.pickle > ./verify.yaml
    # (on lillypilly, that needs to be ../bin/usn.sh --export --db=./database-all.pickle > ./verify.yaml)
    $ diff ./orig.yaml ./verify.yaml    # should have only the USN removed
    # if something went wrong with the deletion, the delete_usn.py script saves a copy
    # of the database before modifying in database-all.pickle-YYYY-MM-DD-HH:MM:SS
  2. Copy the updated USN database by running: ssh people.canonical.com "~ubuntu-security/bin/push-usn-db"

  3. Update your local USN database, remove the USN from the USN website, and track the deployment status by visiting the link printed by the script:

    cd $UCT && ./scripts/fetch-db database-all.pickle.bz2 && ./scripts/publish-usn-to-website -r -d ./database-all.pickle $USN

Local Configuration

  1. Make sure ~/.ubuntu-cve-tracker.conf is fully configured (see the u-c-t README), and set the path to the ubuntu-cve-tracker git tree check-out: UCT=/path/to/ubuntu-cve-tracker

  2. Grab the database.pickle file: cd $UCT; ./scripts/fetch-db database.pickle.bz2

  3. set UQT to the ubuntu-qa-tools tree check-out: UQT=/path/to/ubuntu-qa-tools

  4. Set up ~/.dput.cf with the appropriate Security PPA upload entries:

    • [security]
      fqdn = ppa.launchpad.net
      incoming = ~ubuntu-security/ppa/ubuntu/%(security)s
      login = anonymous
  5. Now you can upload with dput. For example:

    • $ dput -s security:lucid ./*_source.changes  # dry run
      $ dput security:lucid ./*_source.changes

Security Proposed PPA

This ppa is a public PPA which builds packages with only -security enabled. It is used to build packages that need wider testing. Packages can either be tested directly from this PPA, or more often pocket copied into the -proposed. A member of ubuntu-archive is required to copy packages from this PPA to -proposed.

To use the security-proposed PPA:

  1. Set up ~/.dput.cf to have an entry like this for each release:

    • [security-proposed]
      fqdn = ppa.launchpad.net
      incoming = ~ubuntu-security-proposed/ppa/ubuntu/%(security-proposed)s
      login = anonymous
  2. Then can upload with:
    • $ dput -s security-proposed:lucid ./*_source.changes  # dry run
      $ dput security-proposed:lucid ./*_source.changes
      $ umt upload -d security-proposed:xenial # if using the umt tool


CategorySecurityTeam CategoryProcess

SecurityTeam/UpdatePublication (last edited 2024-03-19 04:05:32 by alexmurray)