ReleaseCycle

How the Security Team interacts with the regular Ubuntu Release Cycle.

Security Updates

pre-feature-freeze

  • request syncs for issues fixed in stable releases that are not fixed in the development release, but are fixed in upstream versions.

post-feature-freeze

  • request feature-freeze exceptions for upstream security fixes, or push security patches, depending on extent of upstream changes.

post-beta

  • focus security updates on any updates still missing from devel release that are present in stable updates (see 'Show all out-of-sync CVEs for the devel release' from $UCT/README for assistance)
  • add the following to the announcements section of the ubuntu-security weekly meeting: "Ubuntu <version> is scheduled to release in a few weeks. If you are interested, it would be great if people helped make sure that the Ubuntu CVE Tracker is up to date for the devel release (particularly for universe/multiverse packages."

Regression Testing

pre-Alpha 3

  • run regression tests when any major changes are noticed

Alpha 3

  • run kernel hardening regression tests on multiple architecture/kernel combinations (i386 w/o PAE, i386 w/ PAE, amd64) (test-kernel*.py)
  • run gcc hardening regression tests on amd64 and i386 (test-gcc-security.py)
  • run glibc hardening regression tests on amd64 and i386 (test-glibc-security.py)
  • run other user-space hardening tests (eg PIE) on amd64 and i386 (test-built-binaries.py)

Beta 2

  • run kernel hardening regression tests on multiple architecture/kernel combinations (i386 w/o PAE, i386 w/ PAE, amd64) (test-kernel*.py)
  • run gcc hardening regression tests on amd64 and i386 (test-gcc-security.py)
  • run glibc hardening regression tests on amd64 and i386 (test-glibc-security.py)
  • run other user-space hardening tests (eg PIE) on amd64 and i386 (test-built-binaries.py)
  • run remaining regression tests from qa-regression-testing, updating the tests or reporting bugs against the package
  • run test-rng.py --all --report from qa-regression-testing (takes many days, so may want to spread across multiple machines) and put the results in qa-regression-testing/results/test-rng.py/<release number>. This is likely a good use of EC2 (but keep in mind as of 2009/04/27 it costs ~ $100).

  • run qa-regression-testing/install/get_file_info.sh and review for problems

  • verify guest session is confined

post-rc1

  • run install/get_file_info.sh and record results in qa-regression-testing/install/get_file_info_results (see qa-regression-testing/install/README for details). This also needs to be done on the final images

  • Review supported kernels with the kernel team

EOL Checklist

Infrastructure (EOL)

  • If this is an LTS release that will live on as an ESM release, you need to configure the ESM release before proceeding:
    • Add overlay tracking to all active CVEs. Packages supported in the ESM will receive a copy of the package in the base release's CVE state:

      $ ./scripts/release-cycle-new-overlay.py -ur ${RELEASE}/esm
    • Replace the boilerplate states for the base release with the new overlay release. This is an example command for Precise:

      $ sed -ie 's/^precise_\(.*\): \(.*\)$/precise\/esm_\1: \2/g' \
           -e 's/^#precise_PKG:/#precise\/esm_PKG:/g' \
           active/00boilerplate*
    • Manually update all the boilerplate files according to the ESM support status
      • The package state should be "DNE" for packages that are not supported as part of the ESM release
      • The package state should be "needs-triage" or "needed" for packages that are supported as part of the ESM release
    • update usn.py in the usn-tool tree for proper USN publication
      • update the corresponding value of the get_codename_to_version_dict() function by adding a conditional re-assignment towards the bottom of the function

        • see the inline comments about how to select the right date to compare against
      • pull these changes down on people (this happens automatically, but it will need to happen before a USN publication):

        $ cd /home/ubuntu-security/bzr-pulls/usn-tool/
        $ sudo -H -u ubuntu-security bzr pull
  • from ubuntu-cve-tracker's README section "End of Life":
    • mark any active CVEs for that release with an EOL comment: ./scripts/sync-from-eol.py -WUu -r $RELEASE

    • add release to 'eol_releases' in scripts/cve_lib.py
    • run a CVE retirement cycle:

      $ ./scripts/cve_need_retire -u
    • remove release from boilerplates: sed -i -e '/^<release>_\(.*\): /d' -e '/^#<release>_PKG:/d' ./active/00boilerplate*

    • remove any backport kernels for this release from 00boilerplate.linux
    • from scripts/cve_lib.py, remove:
      • any backport kernels for this release from 'kernel_srcs'
      • the meta kernel and any meta backport kernels from the meta_kernels table
    • update supported_releases and ignored_releases in scripts/generate-oval

  • update ubuntu-cve-tracker/scripts/packages-mirror to lose old release
  • update architectures to include "EOL" footnote in SecurityTeam/FAQ

  • regenerate Security/Features from $UCT/scripts/dump-features --editmoin

  • review and optionally closed bug tasks for EOL releases with $UCT/scripts/report-bugs-for-eol

  • update $QRT/README.testing to remove EOL release
  • update people:~ubuntu-security/.ubuntu-security-tools.conf settings for (used by kernel-abi-check):

    • "release_list" loses EOL release name
  • update security-tools/build-tools/security-fake-sync to ensure LTS releases point to the right place (next_release())

Individuals (EOL)

  • update ~/.ubuntu-security-tools.conf settings for:

    • "release_list" loses EOL release name
  • No longer required with uvt: update ~/.uqt-vm-tools.conf settings for:

    • "vm_release_list" loses EOL release name
  • update debmirror/apt-cacher lists to drop old release
  • delete VMs and schroots from old release
  • adjust sources.list to remove deb and deb-src for old release

    • Then run apt-get update

  • adjust daily build systems

Release Checklist

Infrastructure (release)

  • update $UCT/scripts/cve_lib.py:

    • add the new release in all_releases

    • devel_release is emptied

    • Add release entry and timestamp to release_stamps. (The time stamp can be generated based of the date stamp of the release file rounded up to the next hour; e.g. echo "(($(date -r  /PATH/TO/UBUNTU/ARCHIVE/dists/RELEASE/Release.gpg +%s) / 3600 ) + 1) * 3600" | /usr/bin/bc

    • update release_expectations for the new release, and add a commented out section for the new devel release

    • add release's kernels under MetaKernelTable (based on the kernel team's ABI page) for both this release and LTS any kernels

  • move all active CVEs and boilerplates from "devel" to release state:
    • cd $UCT ; ./scripts/release-cycle-released $RELEASE

  • Update 'releases' in $UCT/scripts/report-todo-recommend to contain the most recent LTS and most recent non-LTS in check for if not opt.all_releases:

  • update supported_releases and ignored_releases in $UCT/scripts/generate-oval

  • Update $UCT/scripts/cve.vim to include the new release.

  • update usn.py in the usn-tool tree for proper USN publication
    • add a new element at the front of the codename_to_version dictionary in the get_codename_to_version_dict() function

    • pull these changes down on people (this happens automatically, but it will need to happen before a USN publication):

      $ cd /home/ubuntu-security/bzr-pulls/usn-tool/
      $ sudo -H -u ubuntu-security bzr pull
  • update architectures in SecurityTeam/FAQ

  • update security-tools/build-tools/security-fake-sync for new stable release (ubuntu_version() and next_release())

  • update people:~ubuntu-security/.ubuntu-security-tools.conf settings for:

    • release_devel is emptied

  • On people, run:

    $ sudo -H -u ubuntu-security /home/ubuntu-security/bin/scripts-diff.sh
    # review the changes, if good:
    $ sudo -H -u ubuntu-security /home/ubuntu-security/bin/scripts-diff.sh --update
  • ask webops in #webops to update the usn-website to have the version for the new release (eg, it should have "Ubuntu <version>"). Eg LP: #773627. This can be done via the '/usn/admin' interface. There shouldn't be an entry for the development release.

  • Update AppArmor

  • Update UncomplicatedFirewall

  • Verify versions for AppArmor in $UCT/scripts/dump-features and regenerate Security/Features from $UCT/scripts/dump-features --editmoin

  • Update list of syscalls for the supported kernel for the new release:

Individuals (release)

  • update ~/.ubuntu-security-tools.conf settings for:

    • release_devel is emptied

  • add security-RELEASE and security-proposed-RELEASE sections to ~/.dput.cf (not normally needed if using recommended dput setup)

  • pull infrastructure changes (from above):
    • usn-tool
    • ubuntu-cve-tracker
    • ubuntu-security-tools
    • ubuntu-qa-tools
  • [optional] download final ISO images
  • [optional] build new clean chroots
  • [optional] build new clean VMs

Devel Opens

Infrastructure (devel)

  • fill in all_releases, flavor_releases, devel_release, release_names and release_expectations in ubuntu-cve-tracker/scripts/cve_lib.py

  • Move all active CVEs and boilerplates from latest release to devel state:
    • ./scripts/release-cycle-devel-opens $LATEST_STABLE_RELEASE

  • add release to non-ports and ports section, as well as the compressed_ext() function, of ubuntu-cve-tracker/scripts/packages-mirror

  • It is helpful to run packages-mirror in the background now as the results are used in several future steps
    • $UCT/scripts/packages-mirror -p -r <new devel> ; $UCT/scripts/packages-mirror -r <new_devel>

  • update security-tools/build-tools/security-fake-sync for new release (next_release())

  • update graphing scripts on people in ~ubuntu-security/graphing (see ~ubuntu-security/graphing/README for details)

    • supported.data.template has the binary package counts. To update:

      • determine the binary package counts ($packages_mirror is set in ~/.ubuntu-cve-tracker.conf):

        $ cd $packages_mirror ; zcat ./dists/$LATEST_STABLE_RELEASE/main/binary-*/Packages.gz ./dists/$LATEST_STABLE_RELEASE/restricted/binary-*/Packages.gz | grep '^Package:' | sort -u | wc -l
      • in the .template file, take the last line of the file and duplicate it at the end of the file
      • in the .template file, uncomment the second to last line (formerly the last line) and add the package counts for the latest stable release
      • in the .template file, update the last line's commented out release name to be the new devel release and adjust counts for EOL releases
      • in the .template file, add the new devel release column header name
      • in the .template file, add a 0 column to all lines (except the last uncommented line), making sure everything is sensible for package counts
    • supported-src.data.template:

      • determine the source package counts:

        $ cd $packages_mirror ; zcat ./dists/$LATEST_STABLE_RELEASE/main/source/Sources.gz ./dists/$LATEST_STABLE_RELEASE/restricted/source/Sources.gz | grep '^Package:' | sort -u | wc -l
      • in the .template file, take the last line of the file and duplicate it at the end of the file
      • in the .template file, uncomment the second to last line (formerly the last line) and add the package counts for the latest stable release
      • in the .template file, update the last line's commented out release name to be the new devel release and adjust counts for EOL releases
      • in the .template file, add the new devel release column header name
      • in the .template file, add a 0 column to all lines (except the last uncommented line), making sure everything is sensible for package counts
    • update.sh REL updated, INDEX incremented by 1 (to match column in the .template files excluding the newly added devel release, since we don't know its package counts yet))

    • add to .bzrignore:

      $NEW_RELEASE-src.data
      $NEW_RELEASE.data
    • commit the changes
  • update people:~ubuntu-security/.ubuntu-security-tools.conf settings for:

    • release_list gains new release name (and any overlay releases, eg 'vivid/stable-phone-overlay', 'xenial/stable-phone-overlay', etc)

    • release_devel gains new release name (without overlay releases)

  • regenerate Security/Features from $UCT/scripts/dump-features

  • regenerate Security/Features/Historical from $UCT/scripts/dump-features --historical

  • update SecurityTeam/KnowledgeBase/AppArmorProfiles

  • create new release in apparmor-profiles and copy profiles over from previous release
  • Make sure lp:ubuntu-qa-tools/responses/update-bug includes the new release
  • update $QRT/README.testing for new development release
  • update ubuntu-security crontab on people to use the new development release for:
    • create-dashboard.sh
      • The argument should be the first letter of the new release name (lowercase)
  • use cd $UCT && ./scripts/sync-from-versions.py -s not-affected. If satisfied with the result, use cd $UCT && ./scripts/sync-from-versions.py -s not-affected -u

  • use cd $UCT && ./scripts/sync-from-versions.py -r $RELEASE -s not-affected. If satisfied with the result, use cd $UCT && ./scripts/sync-from-versions.py -r $RELEASE -s not-affected -u

Individuals (devel)

  • pull infrastructure changes (from above):
    • ubuntu-cve-tracker
    • ubuntu-security-tools
    • ubuntu-qa-tools
    • qa-regression-testing
  • update debmirror/apt-cacher lists to include new release
  • update ~/.ubuntu-security-tools.conf settings for:

    • release_list gains new release name

    • release_devel gains new release name

  • build new VMs and schroots for new release (remembering to update 'vm_release_list' in '~/.uqt-vm-tools.conf' (not needed if using uvt))
    • debootstrap is likely to not yet know about the new dev release and will cause mk-sbuild to fail. Create a symlink in /usr/share/debootstrap/scripts/ that points to the same file as the latest stable release's symlink.

    • install images for the dev release are not likely to exist towards the start of the development cycle. Try using uvt clone to create a new VM based on the latest stable release. After cloning the VM, start the new vm, log in and follow these steps:

      1. Generate new SSH host keys:

        $ sudo rm /etc/ssh/ssh_host_*key*
        $ sudo ssh-keygen -A
      2. Update /etc/apt/sources.list and /etc/apt/sources.list.d/* to replace the latest stable release name with the new dev release name and then update:

        $ sudo apt update
        $ sudo apt dist-upgrade
      3. Shut the VM down and use uvt snapshot in your host environment to preserve your changes

  • update your packages mirror with $UCT/scripts/packages-mirror -r $NEW_RELEASE

  • update your partner packages mirror with $UCT/scripts/packages-mirror -p

  • update your sources.list to have the appropriate deb-src lines for the new release (eg use $UST/build-tools/build-sources-list output in /etc/apt/sources.list.d/ubuntu-security.list)

    • Then run apt-get update

  • [optional] subscribe to $NEW_RELEASE-changes at https://lists.ubuntu.com/mailman/listinfo/$NEW_RELEASE-changes

QA (devel)

For projects where the security team is the upstream maintainer (eg, AppArmor, ufw, ecryptfs):

  • Update the daily build PPA to build for the development release
  • As needed, update the daily build PPA packaging (eg, merge in any changes since the last release)

Click/AppArmor (devel)

  • apparmor-easyprof-ubuntu, ubuntu-core-security and ubuntu-personal-security
    • run sh ./debian/make-new-version.sh <old> <new> which will create symlinks for the new policy to the old previous version

    • increment the package version such that the major and minor versions match that of the new policy directories, and start the micro version at '1'. The policy version should match the release version. Eg, if the apparmor-easyprof-ubuntu policy is currently 15.10.17 and the new version added in the previous step is '16.04', then the new apparmor-easyprof-ubuntu version should be '16.04.1'. See README.source for details.
    • adjust autopkgtests in debian/tests for the new policy
    • add any new tests for the new policy to tests/* as needed
    • adjust QRT/tests/image/unprivileged/apparmor-easyprof-ubuntu as necessary (not usually needed)
  • click-apparmor
    • update 'framework_transforms' in apparmor/click.py for the new release, using the new policy version that was added to apparmor-easyprof-ubuntu
    • add a new 'ubuntu_NNNN_transform()' for the new release (must match what is in 'framework_transforms'
    • add any new autopkgtests for the new 'framework_transforms' and 'ubuntu_NNNN_transform()'
    • add any new tests for the new 'framework_transforms' and 'ubuntu_NNNN_transform()' to test-clicktool.py
    • adjust QRT/tests/image/unprivileged/click-apparmor as necessary (not usually required)
  • click-reviewers-tools
    • add new frameworks/policy to ./data/apparmor-easyprof-ubuntu.json

LTS loses Desktop support

  • generate $UCT/$RELEASE-supported.txt with a list of all the source packages that will get 5 years of support.

  • update $UCT/scripts/cve_lib.py's lts_partial_supported_releases variable to include th LTS.

  • run $UCT/scripts/check-syntax to update all the CVEs to the "ignored" state for the EOL packages.

Debian Release/EOL Checklist

Infrastructure (Release/EOL)

  • Update lp:ubuntu-security-tools for:
    • README needs the proper release_extras list
    • build-tools/build-sources-list has a hard-coded "supported releases" list
  • Update SecurityTeam/BuildEnvironment to mk-sbuild the current releases and document the proper release_extras list

  • Update d2u.py in lp:~ubuntu-security/debian2ubuntu/trunk with new debian release. Will need to pull the bzr tree on people.c.c as well.

    • IMPORTANT: only update the list for new Debian releases since d2u may show pending merges against EOL Debian releases

Individuals (Release/EOL)

  • pull infrastructure changes (from above):
    • ubuntu-security-tools
  • update ~/.ubuntu-security-tools.conf settings for:

    • "release_extras" list gains/drops the release name
  • [optional] create/delete VMs and schroots from the release
  • update your sources.list to add/remove the deb-src lines for the release (eg use $UST/build-tools/build-sources-list output in /etc/apt/sources.list.d/ubuntu-security.list)

    • Then run apt-get update


CategorySecurityTeam

SecurityTeam/ReleaseCycle (last edited 2017-10-27 18:53:17 by tyhicks)