UpdateProcedures

Differences between revisions 79 and 80
Revision 79 as of 2008-10-23 20:41:43
Size: 20269
Editor: sites
Comment: update SiS instructions for local/rookery relocations
Revision 80 as of 2008-10-23 20:56:09
Size: 20275
Editor: sites
Comment:
Deletions are marked like this. Additions are marked like this.
Line 163: Line 163:
 0. Make sure ~/.ubuntu-cve-tracker.conf is fully configured, and set the path to the ubuntu-cve-tracker bzr tree check-out: {{{UCT=/path/to/ubuntu-cve-tracker}}}  0. Make sure {{{~/.ubuntu-cve-tracker.conf}}} is fully configured, and set the path to the ubuntu-cve-tracker bzr tree check-out: {{{UCT=/path/to/ubuntu-cve-tracker}}}
Line 171: Line 171:
 0. Copy the updated USN database by running `ssh rookery "~ubuntu-security/bin/push-usn-db`.  0. Copy the updated USN database by running `ssh rookery "~ubuntu-security/bin/push-usn-db"`

Issues that warrant a security update

We only fix bugs in our stable releases which truly affect overall system security, i. e. which enable an attacker to circumvent the permissions configured on the system, or are a threat to the user's data in any way. Most common examples:

  • Buffer overflow in a server process which allows to crash it (denial of service) and/or to execute attacker provided code (privilege escalation).
  • Insecure temporary file handling which allows race condition and symlink attacks to delete unrelated files with the invoker's privileges.
  • Non-working security-relevant configuration options (e. g. iptables would allow packets which should be blocked, or a server's ACL option does not do the right thing).
  • Less critical bugs (like Denial of Service vulnerabilities in instant messengers or email applications) are also fixed usually, but with lower priority.

Responsibility

The Ubuntu security team (security@ubuntu.com, Launchpad team ubuntu-security) is responsible for all issues that affect source packages in Ubuntu main and restricted. Usually timely updates are provided.

The Ubuntu security team also tracks issues in universe and multiverse. It aims to solve vulnerabilities in the current development release by requesting syncs from Debian. Due to lack of manpower, flaws in stable releases and Ubuntu-modified packages are not fixed by ubuntu-security, but the team will provide assistance in releasing updates which are prepared by other developers.

Preparing an update

  1. Collect as much information about the vulnerability as possible. Use the CVE database, the BugTraq and full-disclosure mailing list (archive), the Debian BTS, upstream bug tracking system, upstream revision control systems, and any other source to collect CAN/CVE numbers, causes, impacts, patches, and possible workarounds.

  2. Prepare an updated package based on the appropriate released source package or the last security update.

    • Make only the minimum changes necessary to fix the bug. This is crucial both for the review process and to avoid introducing bugs or unexpected behavioural changes.
      • In the rare cases when a security update must include a major version update, it should, if possible, go to -proposed first for testing. Any upload by a security team member to -proposed for a security vulnerability will get accepted immediately without ack from ubuntu-sru (similar to how regular security fixes being installed into -updates happen without ack). IMPORTANT: An update that first goes through -proposed should not simply be pocket copied to -updates or -security. It should instead be version bumped with a no change rebuild in the -security queue (this is to avoid dependencies that are only in -updates not being satisfiable when only -security is enabled). There are cases where it may be acceptable to do a pocket copy, but it must be shown that people without -updates installed won't be adversely effected.

    • Prefer packaging the changes as proper patches. When the patch system supports it, please follow PatchTaggingGuidelines and include as much information as possible. If the package does not support patch tagging, please include this information in the changelog (see below for changelog formatting).

    • Update the Maintainer field only if working on Feisty or newer.

  3. Properly test that the package builds and still works. You can refer to QA Regression Testing for scripts and techniques on testing packages.

    • If possible, use publicly available exploits and test cases to verify that the bug is fixed
    • In all cases, verify that the package still functions properly
  4. Determine the version number to be used for the package. This requires some care, because it must be newer than the version being patched, but earlier than any version in the development branch. In general, you should add "0.1" to the Ubuntu revision, similar to Debian's source NMU version numbers. In the case where two Ubuntu releases have the same version number for the package, append the appropriate Ubuntu release version followed by a '.1'.
    • ============================  ===================
      ''Previous version''          ''Security update''
      2.0-2                         2.0-2ubuntu0.1
      2.0-2ubuntu2                  2.0-2ubuntu2.1
      2.0-2ubuntu2.1                2.0-2ubuntu2.2
      2.0-2build1                   2.0-2ubuntu0.1
      2.0                           2.0-0ubuntu0.1
      2.0-2 in two releases         2.0-2ubuntu0.5.04.1 and 2.0-2ubuntu0.5.10.1
      2.0-2ubuntu1 in two releases  2.0-2ubuntu1.5.04.1 and 2.0-2ubuntu1.5.10.1
      ============================  ===================
  5. Create an appropriate changelog entry with a verbose description of the changes and CVE/CAN and other bug number references. Set the distribution [release]-security, e.g. dapper-security or edgy-security, and use a version number that indicates a security update. (Note that "urgency" is not used in Ubuntu, so it should just remain "low".) If there is a corresponding LaunchPad bug open for the issue, include a (LP: #...) section to auto-close the bug.

    • Template:

      [package] ([version]) [release]-security; urgency=low
      
       * SECURITY UPDATE: [how the bad guys could get you] (LP: #num).
         - changed/or/added/file1: [how it was fixed], [thanks to original author]
         - changed/or/added/file2: [how it was fixed], [thanks to original author]
         - http://path/to/patch/source/when/this/url/is/not/in/the/new/debian/patch
         - CVE-YYYY-NNNN
       * ...

      Examples:

      ruby1.8 (1.8.4-1ubuntu1.6) dapper-security; urgency=low
      
        * SECURITY UPDATE: denial of service via resource exhaustion in the REXML
          module (LP: #261459)
          - debian/patches/917_CVE-2008-3790.patch: adjust rexml/document.rb and
            rexml/entity.rb to use expansion limits. Based on upstream patch.
          - CVE-2008-3790
        * SECURITY UPDATE: integer overflow in rb_ary_fill may cause denial of
          service (LP: #246818)
          - debian/patches/918_CVE-2008-2376.patch: adjust array.c to properly
            check argument length. Based on upstream patch.
          - CVE-2008-2376
      mplayer (2:1.0~rc1-0ubuntu13.3) gutsy-security; urgency=low
      
        * SECURITY UPDATE: Multiple integer underflows in MPlayer 1.0_rc2 and
          earlier allow remote attackers to cause a denial of service
          (process termination) and possibly execute arbitrary code via a
          crafted video file that causes the stream_read function to read or
          write arbitrary memory. (LP: #279030)
          - libmpdemux/demux_real.c: Address various integer underflows. Patch from
            oCert.org
          - http://svn.mplayerhq.hu/mplayer/trunk/libmpdemux/demux_real.c?r1=27314&r2=27675
          - CVE-2008-3827
  6. Generate a debdiff against the previous package version, read it to verify that you only did minimal changes and that the changes are indeed correct.
  7. Publish the debdiff in Launchpad for inspection, set the status of the bug to 'In Progress' and make sure that 'ubuntu-security' is subscribed to the bug.
  8. Please indicate the testing performed. Testing might include (but is not limited to) build tests, test suites, PoC, ABI changes, and testing the patched code so that it introduces no regressions. For more information on testing, please see QA Regression Testing.

  9. If not present, please add CVE references to the Launchpad bug using the 'Link to CVE' functionality in Launchpad
  10. Updates prepared directly by the security team should be peer-reviewed. Updates prepared by others will be peer-reviewed by a security team member.
  11. For input on patches, send a note to the MOTU mailing list (ubuntu-motu@lists.ubuntu.com) with a reference to the bug where the debdiff is filed. The security team will upload it at the appropriate time.

  12. If Debian does not yet have a fix, send the package interdiff to the Debian BTS by following Debian/Bugs and Debian/Usertagging (if you use submittodebian, it will take care of all that for you), CC'ing security@debian.org, and adding the Debian bug to the Launchpad bug by using the 'Also affects distribution' functionality in Launchpad.

  13. If the package is in "main" or "restricted", try to write an appropriate "Details" section for an Ubuntu SecurityNotification.

Releasing an update

(For members of the Ubuntu security team)

Upload/Build/Publish

  1. Upload the updated source packages via dput. Notification about failed builds should be automatically sent to security@ubuntu.com. Can use tail -f /srv/ftp.no-name-yet.com/queue/REPORT to see when the packages get accepted.

  2. The finished builds appear in jackass:/srv/ftp.no-name-yet.com/queue/accepted/. cd to this directory.

  3. Run helena to display the pending packages.

  4. Once builds for all supported architectures have finished (e.g. ~kees/bin/check-build firefox), gather the changes files: PKGS=$(ls yourpackage_*changes)

  5. Upload builds to local archive, by running amber 0 $PKGS. (This will also generate an old-style SecurityNotification template). If you built several versions of the package, make sure to push them all out at once, otherwise they will stay in the queue. Optionally append option -n for a dry-run. Answer 'y' to Continue. IMPORTANT: Answer "N" to "Upload to main archive?" question.

  6. After local archive upload, the changes files will have moved to jackass:/srv/ftp.no-name-yet.com/queue/done. cd to this directory.

  7. Verify that there will be no version collision by running ~kees/bin/check-upload-from-changes -s found $PKGS (this should say '0% found'). If any non-orig files are already found, new versions will need to be prepared before continuing.

  8. Upload to the main soyuz archive by running ~kees/bin/merged-upload $PKGS. (This can be run again at a later time if soyuz publication fails.) For kernel updates, inform #canonical-sysadmin that an update will be published soon.

Announce Publication

(for main/restricted publications)

  1. Assign a USN (format is NNN-S, and the following instructions assume $USN has been set as desired):
    1. For a new issue, run USN=$(~kees/bin/get-next-usn) (this reads from and increments ~kees/usn/NEXT)

    2. For an old issue that needs correction or continuation, start with the issue's original USN, keep NNN and increase S. (e.g. original issue was 42-1, updated USN will be 42-2).

    3. For a new issue that affects different software with identical CVEs, run USN=$(~kees/bin/get-next-usn) for the first source package, and then keep NNN and increase S for each additional source package. (e.g. CVE-2008-1693 affected both poppler and koffice, so 603-1 was used for poppler and 603-2 was used for koffice). Please note that different versions of the same software (e.g. emacs21 and emacs22) should not do this, but instead use a single USN with S=1.

  2. Run ~kees/bin/generate-usn $USN $PKGS > ~/new-usn.sh to create a USN template script.

  3. Edit ~/new-usn.sh to include a correct title, summary, action, description, and then limit the binary list to only those affected by the USN. Leave all URLs as-is. Have the description proofread by somebody else.

  4. Run bash ~/new-usn.sh to populate the USN database with the new USN details.

  5. Run ~kees/bin/send-usn-email $USN to generate the template email, which is sent to security@ubuntu.com.

  6. Wait until the packages are actually in the archive; Soyuz security uploads are accepted by LP hourly at :55, and should usually appear on security.ubuntu.com about 40 minutes after that. (Note that uploads at UTC 0355 will skip the 0403 publication run due to the Contents generation job.) Run ~kees/bin/check-upload $USN to verify that the packages have arrived in the archive.

  7. Once packages are in the archive, GPG sign and send the USN email to ubuntu-security-announce@lists.ubuntu.com; CC bugtraq@securityfocus.com and full-disclosure@lists.grok.org.uk.

  8. Approve the USN mail on https://lists.ubuntu.com/mailman/admindb/ubuntu-security-announce. Ensure to reject duplicate mails from you (some list subscribers bounce mails back unmodified).

  9. Create a new USN page for this USN via https://www-admin.ubuntu.com/. Copy&paste the USN email text, but without the file list, wrapped in a "div id=usn" tag, include the cve file path, and a list of all the CVEs.

  10. Copy the updated USN database by running sudo -u ubuntu-security /home/kees/bin/get-usn-db on rookery and ~kees/bin/put-usn-db on jackass.

  11. For large updates (OOo, firefox, kernel, kdebase), ping someone from ArchiveAdministration about doing a pocket-copy from -security to -updates to help reduce the load on security.archive.com.

  12. Check for any outstanding LP bugs tied to the CVEs that are resolved with the USN. https://bugs.launchpad.net/bugs/cve/YYYY-NNNN

Upload/Build/Publish (SiS)

  1. Upload the updated source packages via dput to each release's security PPA target. Notification about failed builds should be automatically sent to security@ubuntu.com.

    • [security-dapper]
      fqdn = ppa.launchpad.net
      incoming = ~ubuntu-security/ubuntu/dapper
      login = anonymous
      
      [security-feisty]
      fqdn = ppa.launchpad.net
      incoming = ~ubuntu-security/ubuntu/feisty
      login = anonymous
      
      [security-gutsy]
      fqdn = ppa.launchpad.net
      incoming = ~ubuntu-security/ubuntu/gutsy
      login = anonymous
      
      [security-hardy]
      fqdn = ppa.launchpad.net
      incoming = ~ubuntu-security/ubuntu/hardy
      login = anonymous
  2. Wait for the finished builds on all supported architectures to finish and appear at the Ubuntu Security PPA. (command line tool needed!)

  3. Request unembargoing from a LOSA or cprov (ssh forced-command needed!)

Announce Publication (SiS)

(for main/restricted publications)

  1. Assign a USN (format is NNN-S, and the following instructions assume $USN has been set as desired):
    1. For a new issue, run USN=$(ssh rookery "~ubuntu-security/bin/get-next-usn") (this reads from and increments ~ubuntu-security/usn/NEXT)

    2. For an old issue that needs correction or continuation, start with the issue's original USN, keep NNN and increase S. (e.g. original issue was 42-1, updated USN will be 42-2).

    3. For a new issue that affects different software with identical CVEs, run USN=$(ssh rookery "~ubuntu-security/bin/get-next-usn") for the first source package, and then keep NNN and increase S for each additional source package. (e.g. CVE-2008-1693 affected both poppler and koffice, so 603-1 was used for poppler and 603-2 was used for koffice). Please note that different versions of the same software (e.g. emacs21 and emacs22) should not do this, but instead use a single USN with S=1.

  2. Make sure ~/.ubuntu-cve-tracker.conf is fully configured, and set the path to the ubuntu-cve-tracker bzr tree check-out: UCT=/path/to/ubuntu-cve-tracker

  3. Run in $UCT/scripts/sis-changes --download /tmp/pending SRCPKG && cd /tmp/pending && $UCT/scripts/sis-generate-usn $USN *.changes > ~/new-usn.sh to create a USN template script (where "SRCPKG" is the source package name being update).

  4. Edit ~/new-usn.sh to include a correct title, summary, action, description, and then limit the binary list to only those affected by the USN. Leave all URLs as-is. Have the description proofread by somebody else.

  5. Run bash ~/new-usn.sh to populate the USN database with the new USN details, and will generate the template email, which is sent to security@ubuntu.com.

  6. Wait until the packages are actually in the archive; Soyuz security uploads are accepted by LP hourly at :55, and should usually appear on security.ubuntu.com about 40 minutes after that. (Note that uploads at UTC 0355 will skip the 0403 publication run due to the Contents generation job.) Run $UCT/scripts/check-upload $USN to verify that the packages have arrived in the archive.

  7. Once packages are in the archive, GPG sign and send the USN email to ubuntu-security-announce@lists.ubuntu.com; CC bugtraq@securityfocus.com and full-disclosure@lists.grok.org.uk.

  8. Approve the USN mail on https://lists.ubuntu.com/mailman/admindb/ubuntu-security-announce. Ensure to reject duplicate mails from you (some list subscribers bounce mails back unmodified).

  9. Create a new USN page for this USN via https://www-admin.ubuntu.com/. Copy&paste the USN email text, but without the file list, wrapped in a "div id=usn" tag, include the cve file path, and a list of all the CVEs.

  10. Copy the updated USN database by running ssh rookery "~ubuntu-security/bin/push-usn-db"

  11. For large updates (OOo, firefox, kernel, kdebase), ping an archive admin about doing a pocket-copy from -security to -updates to help reduce the load on security.archive.com. (ssh forced-command needed!)

  12. Check for any outstanding LP bugs tied to the CVEs that are resolved with the USN. https://bugs.launchpad.net/bugs/cve/YYYY-NNNN

  13. Delete PKGS from Security PPA.

Publication of Shared orig.tar.gz Source Packages

If a package uses the same orig.tar.gz between releases (eg, dapper, edgy and feisty mozilla-thunderbird all use the same orig.tar.gz), then need to make sure that the first one (in this case, dapper) gets built/published before the others (eg, edgy and feisty). This must happen on both dak and soyuz. Therefore, the order of operations becomes:

  1. Perform all steps of Upload/Build/Publish for the earliest release that uses the shared orig.tar.gz (eg, dapper)
  2. Perform all steps up to but not including the upload to soyuz of Upload/Build/Publish for all other releases that use the shared orig.tar.gz (eg, edgy and feisty)
  3. Wait until the packages for the earliest release that uses the shared orig.tar.gz (eg, dapper) appear on security.ubuntu.com. You can cd to jackass:/srv/ftp.no-name-yet.com/queue/done and run ~kees/bin/check-upload-from-changes $PKGS where PKGS contains only the *changes files from the earliest release (eg, dapper). This command should say 100% found.

  4. Once the packages from the earliest release that uses the shared orig.tar.gz (eg dapper) are in the archive, upload packages to soyuz for all other releases that use the shared orig.tar.gz (eg, edgy and feisty)
  5. Continue with announce publication as required.

Editing a Published USN

  1. Edit the page for this USN via https://www-admin.ubuntu.com/.

  2. Update the USN database on jackass with ~kees/bin/edit-usn NNN-S, where NNN-S corresponds to the edited USN, eg 582-2.

  3. Copy the updated USN database by running sudo -u ubuntu-security /home/kees/bin/get-usn-db on rookery and ~kees/bin/put-usn-db on jackass.

Regressions

In the case of regressions caused by security updates, please follow the SRU regression policy.


CategoryProcess

SecurityTeam/UpdateProcedures (last edited 2019-09-16 14:57:57 by seth-arnold)