UpdateProcedures

Revision 68 as of 2008-09-30 20:11:25

Clear message

Issues that warrant a security update

We only fix bugs in our stable releases which truly affect overall system security, i. e. which enable an attacker to circumvent the permissions configured on the system, or are a threat to the user's data in any way. Most common examples:

  • Buffer overflow in a server process which allows to crash it (denial of service) and/or to execute attacker provided code (privilege escalation).
  • Insecure temporary file handling which allows race condition and symlink attacks to delete unrelated files with the invoker's privileges.
  • Non-working security-relevant configuration options (e. g. iptables would allow packets which should be blocked, or a server's ACL option does not do the right thing).
  • Less critical bugs (like Denial of Service vulnerabilities in instant messengers or email applications) are also fixed usually, but with lower priority.

Responsibility

The Ubuntu security team (security@ubuntu.com, Launchpad team ubuntu-security) is responsible for all issues that affect source packages in Ubuntu main and restricted. Usually timely updates are provided.

The Ubuntu security team also tracks issues in universe and multiverse. It aims to solve vulnerabilities in the current development release by requesting syncs from Debian. Due to lack of manpower, flaws in stable releases and Ubuntu-modified packages are not fixed by ubuntu-security, but the team will provide assistance in releasing updates which are prepared by other developers.

Preparing an update

  1. Collect as much information about the vulnerability as possible. Use the CVE database, the BugTraq and full-disclosure mailing list (archive), the Debian BTS, upstream bug tracking system, upstream revision control systems, and any other source to collect CAN/CVE numbers, causes, impacts, patches, and possible workarounds.

  2. Prepare an updated package based on the appropriate released source package or the last security update.

    • Make only the minimum changes necessary to fix the bug. This is crucial both for the review process and to avoid introducing bugs or unexpected behavioural changes.
      • In the rare cases when a security update must include a major version update, it should, if possible, go to -proposed first for testing. Any upload by a security team member to -proposed for a security vulnerability will get accepted immediately without ack from ubuntu-sru (similar to how regular security fixes being installed into -updates happen without ack). IMPORTANT: An update that first goes through -proposed should not simply be pocket copied to -updates or -security. It should instead be version bumped with a no change rebuild in the -security queue (this is to avoid dependencies that are only in -updates not being satisfiable when only -security is enabled). There are cases where it may be acceptable to do a pocket copy, but it must be shown that people without -updates installed won't be adversely effected.

    • Prefer packaging the changes as proper patches

    • Update the Maintainer field only if working on Feisty or newer.

  3. Properly test that the package builds and still works. You can refer to QA Regression Testing for scripts and techniques on testing packages.

    • If possible, use publicly available exploits and test cases to verify that the bug is fixed
    • In all cases, verify that the package still functions properly
  4. Determine the version number to be used for the package. This requires some care, because it must be newer than the version being patched, but earlier than any version in the development branch. In general, you should add "0.1" to the Ubuntu revision, similar to Debian's source NMU version numbers. In the case where two Ubuntu releases have the same version number for the package, append the appropriate Ubuntu release version followed by a '.1'.
    • =====================  ===================
      ''Previous version''   ''Security update''
      2.0-2                  2.0-2ubuntu0.1
      2.0-2ubuntu2           2.0-2ubuntu2.1
      2.0-2ubuntu2.1         2.0-2ubuntu2.2
      2.0-2build1            2.0-2ubuntu0.1
      2.0                    2.0-0ubuntu0.1
      2.0-2 in two releases  2.0-2ubuntu0.5.04.1 and 2.0-2ubuntu0.5.10.1
      =====================  ===================
  5. Create an appropriate changelog entry with a verbose description of the changes and CVE/CAN and other bug number references. Set the distribution to dapper-security (or edgy-security or whatever) and use a version number that indicates a security update. If there is a corresponding LaunchPad bug open for the issue, include a (Fixes LP: #...) section to auto-close the bug.

    • Template:

      • [package] ([version]) [release]-security; urgency=low
        
          * SECURITY UPDATE: [what can happen when it misbehaves]
          * [patches added, files changed]: [what changed].  [patch origin]. (Fixes LP: #...)
          * References
            [bug or patch URL if useful]
            [CVEs if available]

      Real-world Examples:

      • rpm (4.4.1-5ubuntu2.1) dapper-security; urgency=low
        
          * SECURITY UPDATE: heap overflow in query report could lead to arbitrary
            code execution.
          * Add 'debian/patches/99_query_heap_protection.diff': validate message
            length.  Patch from upstream CVS, applied inline.
          * References
            https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=212833
            CVE-2006-5466
        openldap2.2 (2.2.26-5ubuntu2.2) dapper-security; urgency=low
        
          * SECURITY UPDATE: Denial of service possible with a crafted remote
            LDAP BIND request due to an assert failure.
          * libraries/libldap/getdn.c: check for string end conditions, as done
            in upstream CVS.
          * References
            CVE-2006-5779
  6. Generate a debdiff against the previous package version, read it to verify that you only did minimal changes and that the changes are indeed correct.
  7. Publish the debdiff in Launchpad for inspection, set the status of the bug to 'In Progress' and make sure that 'ubuntu-security' is subscribed to the bug.
  8. Please indicate the testing performed. Testing might include (but is not limited to) build tests, test suites, PoC, ABI changes, and testing the patched code so that it introduces no regressions. For more information on testing, please see QA Regression Testing.

  9. If not present, please add CVE references to the Launchpad bug using the 'Link to CVE' functionality in Launchpad
  10. Updates prepared directly by the security team should be peer-reviewed. Updates prepared by others will be peer-reviewed by a security team member.
  11. For input on patches, send a note to the MOTU mailing list (ubuntu-motu@lists.ubuntu.com) with a reference to the bug where the debdiff is filed. The security team will upload it at the appropriate time.

  12. If Debian does not yet have a fix, send the package interdiff to the Debian BTS (create a new bug following Debian/Usertagging if necessary), CC'ing security@debian.org

  13. If the package is in "main" or "restricted", try to write an appropriate "Details" section for an Ubuntu SecurityNotification.

Releasing an update

(For members of the Ubuntu security team)

Upload/Build/Publish

  1. Upload the updated source packages via dput. Notification about failed builds should be automatically sent to security@ubuntu.com. Can use tail -f /srv/ftp.no-name-yet.com/queue/REPORT to see when the packages get accepted.

  2. The finished builds appear in jackass:/srv/ftp.no-name-yet.com/queue/accepted/. cd to this directory.

  3. Run helena to display the pending packages.

  4. Once builds for all supported architectures have finished (e.g. ~kees/bin/check-build firefox), gather the changes files: PKGS=$(ls yourpackage_*changes)

  5. Upload builds to local archive, by running amber 0 $PKGS. (This will also generate an old-style SecurityNotification template). If you built several versions of the package, make sure to push them all out at once, otherwise they will stay in the queue. Optionally append option -n for a dry-run. Answer 'y' to Continue. IMPORTANT: Answer "N" to "Upload to main archive?" question.

  6. After local archive upload, the changes files will have moved to jackass:/srv/ftp.no-name-yet.com/queue/done. cd to this directory.

  7. Verify that there will be no version collision by running ~kees/bin/check-upload-from-changes -s found $PKGS (this should say '0% found'). If any non-orig files are already found, new versions will need to be prepared before continuing.

  8. Upload to the main soyuz archive by running ~kees/bin/merged-upload $PKGS. (This can be run again at a later time if soyuz publication fails.) For kernel updates, inform #canonical-sysadmin that an update will be published soon.

Announce Publication

(for main/restricted publications)

  1. Assign a USN (format is NNN-S, and the following instructions assume $USN has been set as desired):
    1. For a new issue, run USN=$(~kees/bin/get-next-usn) (this reads from and increments ~kees/usn/NEXT)

    2. For an old issue that needs correction or continuation, start with the issue's original USN, keep NNN and increase S. (e.g. original issue was 42-1, updated USN will be 42-2).

    3. For a new issue that affects different software with identical CVEs, run USN=$(~kees/bin/get-next-usn) for the first source package, and then keep NNN and increase S for each additional source package. (e.g. CVE-2008-1693 affected both poppler and koffice, so 603-1 was used for poppler and 603-2 was used for koffice). Please note that different versions of the same software (e.g. emacs21 and emacs22) should not do this, but instead use a single USN with S=1.

  2. Run ~kees/bin/generate-usn $USN $PKGS > ~/new-usn.sh to create a USN template script.

  3. Edit ~/new-usn.sh to include a correct title, summary, action, description, and then limit the binary list to only those affected by the USN. Leave all URLs as-is. Have the description proofread by somebody else.

  4. Run bash ~/new-usn.sh to populate the USN database with the new USN details.

  5. Run ~kees/bin/send-usn-email $USN to generate the template email, which is sent to security@ubuntu.com.

  6. Wait until the packages are actually in the archive; Soyuz security uploads are accepted by LP hourly at :55, and should usually appear on security.ubuntu.com about 40 minutes after that. (Note that uploads at UTC 0355 will skip the 0403 publication run due to the Contents generation job.) Run ~kees/bin/check-upload $USN to verify that the packages have arrived in the archive.

  7. Once packages are in the archive, GPG sign and send the USN email to ubuntu-security-announce@lists.ubuntu.com; CC bugtraq@securityfocus.com and full-disclosure@lists.grok.org.uk.

  8. Approve the USN mail on https://lists.ubuntu.com/mailman/admindb/ubuntu-security-announce. Ensure to reject duplicate mails from you (some list subscribers bounce mails back unmodified).

  9. Create a new USN page for this USN via https://www-admin.ubuntu.com/. Copy&paste the USN email text, but without the file list, wrapped in a "div id=usn" tag, include the cve file path, and a list of all the CVEs.

  10. Copy the updated USN database by running sudo -u ubuntu-security /home/kees/bin/get-usn-db on rookery and ~kees/bin/put-usn-db on jackass.

  11. For large updates (OOo, firefox, kernel, kdebase), ping someone from ArchiveAdministration about doing a pocket-copy from -security to -updates to help reduce the load on security.archive.com.

  12. Check for any outstanding LP bugs tied to the CVEs that are resolved with the USN. https://bugs.launchpad.net/bugs/cve/YYYY-NNNN

Upload/Build/Publish (SiS)

  1. Upload the updated source packages via dput to each release's security PPA target. Notification about failed builds should be automatically sent to security@ubuntu.com.

    • [security-dapper]
      fqdn = ppa.launchpad.net
      incoming = ~ubuntu-security/ubuntu/dapper
      login = anonymous
      
      [security-feisty]
      fqdn = ppa.launchpad.net
      incoming = ~ubuntu-security/ubuntu/feisty
      login = anonymous
      
      [security-gutsy]
      fqdn = ppa.launchpad.net
      incoming = ~ubuntu-security/ubuntu/gutsy
      login = anonymous
      
      [security-hardy]
      fqdn = ppa.launchpad.net
      incoming = ~ubuntu-security/ubuntu/hardy
      login = anonymous
  2. Wait for the finished builds on all supported architectures to finish and appear at the Ubuntu Security PPA. (command line tool needed!)

  3. Request unembargoing from ArchiveAdministration (ssh forced-command needed!)

Announce Publication (SiS)

(for main/restricted publications)

  1. Assign a USN (format is NNN-S, and the following instructions assume $USN has been set as desired):
    1. For a new issue, run USN=$(~kees/bin/get-next-usn) (this reads from and increments ~kees/usn/NEXT)

    2. For an old issue that needs correction or continuation, start with the issue's original USN, keep NNN and increase S. (e.g. original issue was 42-1, updated USN will be 42-2).

    3. For a new issue that affects different software with identical CVEs, run USN=$(~kees/bin/get-next-usn) for the first source package, and then keep NNN and increase S for each additional source package. (e.g. CVE-2008-1693 affected both poppler and koffice, so 603-1 was used for poppler and 603-2 was used for koffice). Please note that different versions of the same software (e.g. emacs21 and emacs22) should not do this, but instead use a single USN with S=1.

  2. Run ~kees/bin/sis-pull-gen $USN > ~/new-usn.sh to create a USN template script. This will prompt you to run the appropriate sis-changes command locally and paste in the changes and dsc file URL. The script sis-changes is in ubuntu-cve-tracker bzr. Each version of a given source package must be specified: scripts/sis-changes ubuntu-security SRCPKG VERSION VERSION...

  3. Edit ~/new-usn.sh to include a correct title, summary, action, description, and then limit the binary list to only those affected by the USN. Leave all URLs as-is. Have the description proofread by somebody else.

  4. Run bash ~/new-usn.sh to populate the USN database with the new USN details.

  5. Run ~kees/bin/send-usn-email $USN to generate the template email, which is sent to security@ubuntu.com.

  6. Wait until the packages are actually in the archive; Soyuz security uploads are accepted by LP hourly at :55, and should usually appear on security.ubuntu.com about 40 minutes after that. (Note that uploads at UTC 0355 will skip the 0403 publication run due to the Contents generation job.) Run ~kees/bin/check-upload $USN to verify that the packages have arrived in the archive.

  7. Once packages are in the archive, GPG sign and send the USN email to ubuntu-security-announce@lists.ubuntu.com; CC bugtraq@securityfocus.com and full-disclosure@lists.grok.org.uk.

  8. Approve the USN mail on https://lists.ubuntu.com/mailman/admindb/ubuntu-security-announce. Ensure to reject duplicate mails from you (some list subscribers bounce mails back unmodified).

  9. Create a new USN page for this USN via https://www-admin.ubuntu.com/. Copy&paste the USN email text, but without the file list, wrapped in a "div id=usn" tag, include the cve file path, and a list of all the CVEs.

  10. Copy the updated USN database by running sudo -u ubuntu-security /home/kees/bin/get-usn-db on rookery and ~kees/bin/put-usn-db on jackass.

  11. For large updates (OOo, firefox, kernel, kdebase), ping someone from ArchiveAdministration about doing a pocket-copy from -security to -updates to help reduce the load on security.archive.com. (ssh forced-command needed!)

  12. Check for any outstanding LP bugs tied to the CVEs that are resolved with the USN. https://bugs.launchpad.net/bugs/cve/YYYY-NNNN

  13. Delete PKGS from Security PPA.

Publication of Shared orig.tar.gz Source Packages

If a package uses the same orig.tar.gz between releases (eg, dapper, edgy and feisty mozilla-thunderbird all use the same orig.tar.gz), then need to make sure that the first one (in this case, dapper) gets built/published before the others (eg, edgy and feisty). This must happen on both dak and soyuz. Therefore, the order of operations becomes:

  1. Perform all steps of Upload/Build/Publish for the earliest release that uses the shared orig.tar.gz (eg, dapper)
  2. Perform all steps up to but not including the upload to soyuz of Upload/Build/Publish for all other releases that use the shared orig.tar.gz (eg, edgy and feisty)
  3. Wait until the packages for the earliest release that uses the shared orig.tar.gz (eg, dapper) appear on security.ubuntu.com. You can cd to jackass:/srv/ftp.no-name-yet.com/queue/done and run ~kees/bin/check-upload-from-changes $PKGS where PKGS contains only the *changes files from the earliest release (eg, dapper). This command should say 100% found.

  4. Once the packages from the earliest release that uses the shared orig.tar.gz (eg dapper) are in the archive, upload packages to soyuz for all other releases that use the shared orig.tar.gz (eg, edgy and feisty)
  5. Continue with announce publication as required.

Editing a Published USN

  1. Edit the page for this USN via https://www-admin.ubuntu.com/.

  2. Update the USN database on jackass with ~kees/bin/edit-usn NNN-S, where NNN-S corresponds to the edited USN, eg 582-2.

  3. Copy the updated USN database by running sudo -u ubuntu-security /home/kees/bin/get-usn-db on rookery and ~kees/bin/put-usn-db on jackass.

Regressions

In the case of regressions caused by security updates, please follow the SRU regression policy.


CategoryProcess