ReleaseCycle

Revision 179 as of 2019-10-30 15:55:58

Clear message

How the Security Team interacts with the regular Ubuntu Release Cycle.

Security Updates

pre-feature-freeze

  • request syncs for issues fixed in stable releases that are not fixed in the development release, but are fixed in upstream versions.

post-feature-freeze

  • request feature-freeze exceptions for upstream security fixes, or push security patches, depending on extent of upstream changes.

post-beta

  • focus security updates on any updates still missing from devel release that are present in stable updates (see 'Show all out-of-sync CVEs for the devel release' from $UCT/README for assistance)
  • add the following to the announcements section of the ubuntu-security weekly meeting: "Ubuntu <version> is scheduled to release in a few weeks. If you are interested, it would be great if people helped make sure that the Ubuntu CVE Tracker is up to date for the devel release (particularly for universe/multiverse packages."

Regression Testing

pre-Alpha 3

  • run regression tests when any major changes are noticed

Alpha 3

  • run kernel hardening regression tests on multiple architecture/kernel combinations (i386 w/o PAE, i386 w/ PAE, amd64) (test-kernel*.py)
  • run gcc hardening regression tests on amd64 and i386 (test-gcc-security.py)
  • run glibc hardening regression tests on amd64 and i386 (test-glibc-security.py)
  • run other user-space hardening tests (eg PIE) on amd64 and i386 (test-built-binaries.py)

Beta 2

  • run kernel hardening regression tests on multiple architecture/kernel combinations (i386 w/o PAE, i386 w/ PAE, amd64) (test-kernel*.py)
  • run gcc hardening regression tests on amd64 and i386 (test-gcc-security.py)
  • run glibc hardening regression tests on amd64 and i386 (test-glibc-security.py)
  • run other user-space hardening tests (eg PIE) on amd64 and i386 (test-built-binaries.py)
  • run remaining regression tests from qa-regression-testing, updating the tests or reporting bugs against the package
  • run test-rng.py --all --report from qa-regression-testing (takes many days, so may want to spread across multiple machines) and put the results in qa-regression-testing/results/test-rng.py/<release number>. This is likely a good use of EC2 (but keep in mind as of 2009/04/27 it costs ~ $100).

  • run qa-regression-testing/install/get_file_info.sh and review for problems

  • verify guest session is confined

post-rc1

  • run install/get_file_info.sh and record results in qa-regression-testing/install/get_file_info_results (see qa-regression-testing/install/README for details). This also needs to be done on the final images

  • Review supported kernels with the kernel team

EOL Checklist

Infrastructure (EOL)

  • If this is an LTS release that will live on as an ESM release, you need to configure the ESM release before proceeding:
    • Add ESM release to all_releases and release_names sections of cve_lib.py
    • Add overlay tracking to all active CVEs. Packages supported in the ESM will receive a copy of the package in the base release's CVE state:

      $ ./scripts/release-cycle-new-overlay.py -ur ${RELEASE}/esm
    • Replace the boilerplate states for the base release with the new overlay release. This is an example command for Precise:

      $ sed -i -e 's/^precise_\(.*\): \(.*\)$/precise\/esm_\1: \2/g' \
           -e 's/^#precise_PKG:/#precise\/esm_PKG:/g' \
           active/00boilerplate*
    • Manually update all the boilerplate files according to the ESM support status
      • The package state should be "DNE" for packages that are not supported as part of the ESM release
      • The package state should be "needs-triage" or "needed" for packages that are supported as part of the ESM release
    • update usn.py in the usn-tool tree for proper USN publication
      • update the corresponding value of the get_codename_to_version_dict() function by adding a conditional re-assignment towards the bottom of the function

        • see the inline comments about how to select the right date to compare against
      • pull these changes down on people (this happens automatically, but it will need to happen before a USN publication):

        $ cd /home/ubuntu-security/git-pulls/usn-tool/
        $ sudo -H -u ubuntu-security git pull --ff-only
  • from ubuntu-cve-tracker's README section "End of Life":
    • mark any active CVEs for that release with an EOL comment: ./scripts/sync-from-eol.py -WUu -r $RELEASE

    • add release to 'eol_releases' in scripts/cve_lib.py
    • run a CVE retirement cycle:

      $ ./scripts/cve_need_retire -u
    • remove release from boilerplates: sed -i -e '/^<release>_\(.*\): /d' -e '/^#<release>_PKG:/d' ./active/00boilerplate*

    • remove any backport kernels for this release from 00boilerplate.linux
    • from scripts/cve_lib.py, remove:
      • any backport kernels for this release from 'kernel_srcs'
      • the meta kernel and any meta backport kernels from the meta_kernels table
    • update supported_releases and ignored_releases in scripts/generate-oval

  • update ubuntu-cve-tracker/scripts/packages-mirror to lose old release
  • update architectures to include "EOL" footnote in SecurityTeam/FAQ

  • regenerate Security/Features from $UCT/scripts/dump-features --editmoin

  • review and optionally closed bug tasks for EOL releases with $UCT/scripts/report-bugs-for-eol

  • update $QRT/README.testing to remove EOL release
  • update people:~ubuntu-security/.ubuntu-security-tools.conf settings for (used by kernel-abi-check):

    • "release_list" loses EOL release name
  • update security-tools/build-tools/security-fake-sync to ensure LTS releases point to the right place (next_release())

Individuals (EOL)

  • update ~/.ubuntu-security-tools.conf settings for:

    • "release_list" loses EOL release name
    • if this is an LTS release that will live on as an ESM release, then you might want to add RELEASE/esm (where RELEASE is the release name) to release_list.
  • No longer required with uvt: update ~/.uqt-vm-tools.conf settings for:

    • "vm_release_list" loses EOL release name
  • update debmirror/apt-cacher lists to drop old release
  • delete VMs and schroots from old release
    • if this is an LTS release that will live on as an ESM release, then you will probably want to create schroot and VM for it, please check here and here

  • adjust sources.list to remove deb and deb-src for old release

    • if this is an LTS release that will live on as an ESM release, then add ESM ppa to sources.list
    • Then run apt-get update

  • adjust daily build systems

Release Checklist

Infrastructure (release)

  • update $UCT/scripts/cve_lib.py:

    • add the new release in all_releases

    • devel_release is emptied

    • Add release entry and timestamp to release_stamps. (The time stamp can be generated based of the date stamp of the release file rounded up to the next hour; e.g. echo "(($(date -r  /PATH/TO/UBUNTU/ARCHIVE/dists/RELEASE/Release.gpg +%s) / 3600 ) + 1) * 3600" | /usr/bin/bc

    • update release_expectations for the new release, and add a commented out section for the new devel release

    • add release's kernels under MetaKernelTable (based on the kernel team's meta info file) for both this release and any LTS kernels

  • move all active CVEs and boilerplates from "devel" to release state:
    • cd $UCT ; ./scripts/release-cycle-released $RELEASE

      • Note that this also touchs the emabargoed tree, so will need to commit and push there
  • Update 'releases' in $UCT/scripts/report-todo-recommend to contain the most recent LTS and most recent non-LTS in check for if not opt.all_releases:

  • update supported_releases and ignored_releases in $UCT/scripts/generate-oval

  • Update $UCT/scripts/cve.vim, $UCT/scripts/cve-mode.el and $UCT/scripts/uct.el to include the new release.

  • update usn.py in the usn-tool tree for proper USN publication
    • add a new element at the front of the codename_to_version dictionary in the get_codename_to_version_dict() function

    • pull these changes down on people (this happens automatically, but it will need to happen before a USN publication):

      $ cd /home/ubuntu-security/git-pulls/usn-tool/
      $ sudo -H -u ubuntu-security git pull --ff-only
  • update RELEASES etc in kpis/common.sh in the embrgoed tree to include the new release

    • ensure to also add the corresponding new series to the relevant graphs in the Grafana dashboard
  • update architectures in SecurityTeam/FAQ

  • update security-tools/build-tools/security-fake-sync for new stable release (ubuntu_version() and next_release())

  • update people:~ubuntu-security/.ubuntu-security-tools.conf settings for:

    • release_devel is emptied

  • On people, run:

    $ sudo -H -u ubuntu-security /home/ubuntu-security/bin/scripts-diff.sh
    # review the changes, if good:
    $ sudo -H -u ubuntu-security /home/ubuntu-security/bin/scripts-diff.sh --update

  • Update AppArmor

  • Update UncomplicatedFirewall

  • Verify versions for AppArmor in $UCT/scripts/dump-features and regenerate Security/Features from $UCT/scripts/dump-features --editmoin

  • Update list of syscalls for the supported kernel for the new release:
  • update SecurityTeam/KnowledgeBase/AppArmorProfiles

    • eg, https://codesearch.debian.net/search?q=apparmor.d&literal=1 and look for new stuff in Ubuntu or...

    • if have local mirror, use 'for-archive' from UST with the following:

      export GREP="egrep"; do for comp in main universe multiverse; do $HOME/bin/for-archive-tools/for-archive $archive_mirror/dists/$RELEASE/$comp/binary-amd64/Packages.gz $archive_mirror $HOME/bin/for-archive-tools/unpack-list 'apparmor.d'; done | tee ~/apparmor.d-$RELEASE.log

Individuals (release)

  • update ~/.ubuntu-security-tools.conf settings for:

    • release_devel is emptied

  • add security-RELEASE and security-proposed-RELEASE sections to ~/.dput.cf (not normally needed if using recommended dput setup)

  • pull infrastructure changes (from above):
    • usn-tool
    • ubuntu-cve-tracker
    • ubuntu-security-tools
    • ubuntu-qa-tools
  • VIM users should copy the new $UCT/scripts/cve.vim to ~/.vim/syntax/

  • [optional] download final ISO images
  • [optional] build new clean chroots
  • [optional] build new clean VMs

Devel Opens

Infrastructure (devel)

  • fill in all_releases, flavor_releases, devel_release, release_names and release_expectations in ${UCT}/scripts/cve_lib.py

  • Move all active CVEs and boilerplates from latest release to devel state:
    • ./scripts/release-cycle-devel-opens $LATEST_STABLE_RELEASE

  • add release to non-ports and ports section, as well as the compressed_ext() function, of ${UCT}/scripts/packages-mirror

  • It is helpful to run packages-mirror in the background now as the results are used in several future steps
    • $UCT/scripts/packages-mirror -p -r <new devel> ; $UCT/scripts/packages-mirror -r <new_devel>

  • update security-tools/build-tools/security-fake-sync for new release (next_release())

  • update graphing scripts on people in ~ubuntu-security/graphing (see ~ubuntu-security/graphing/README for details)

    • supported.data.template has the binary package counts. To update:

      • determine the binary package counts ($packages_mirror is set in ~/.ubuntu-cve-tracker.conf):

        $ cd $packages_mirror ; zcat ./dists/$LATEST_STABLE_RELEASE/main/binary-*/Packages.gz ./dists/$LATEST_STABLE_RELEASE/restricted/binary-*/Packages.gz | grep '^Package:' | sort -u | wc -l
      • in the .template file, take the last line of the file and duplicate it at the end of the file
      • in the .template file, uncomment the second to last line (formerly the last line) and add the package counts for the latest stable release
      • in the .template file, update the last line's commented out release name to be the new devel release and adjust counts for EOL releases
      • in the .template file, add the new devel release column header name
      • in the .template file, add a 0 column to all lines (except the last uncommented line), making sure everything is sensible for package counts
    • supported-src.data.template:

      • determine the source package counts:

        $ cd $packages_mirror ; zcat ./dists/$LATEST_STABLE_RELEASE/main/source/Sources.gz ./dists/$LATEST_STABLE_RELEASE/restricted/source/Sources.gz | grep '^Package:' | sort -u | wc -l
      • in the .template file, take the last line of the file and duplicate it at the end of the file
      • in the .template file, uncomment the second to last line (formerly the last line) and add the package counts for the latest stable release
      • in the .template file, update the last line's commented out release name to be the new devel release and adjust counts for EOL releases
      • in the .template file, add the new devel release column header name
      • in the .template file, add a 0 column to all lines (except the last uncommented line), making sure everything is sensible for package counts
    • update.sh:

      • REL updated to dev release name

      • RELTAG gets the version of the newly opened dev release (e.g. 18.10)

      • INDEX incremented by 1 (to match column in the .template files excluding the newly added devel release, since we don't know its package counts yet))

    • add to .bzrignore:

      $NEW_RELEASE-src.data
      $NEW_RELEASE.data
    • commit the changes
  • update people:~ubuntu-security/.ubuntu-security-tools.conf settings for:

    • release_list gains new release name (and any overlay releases, eg 'vivid/stable-phone-overlay', 'xenial/stable-phone-overlay', etc)

    • release_devel gains new release name (without overlay releases)

  • regenerate Security/Features from $UCT/scripts/dump-features

  • regenerate Security/Features/Historical from $UCT/scripts/dump-features --historical

  • create new release in apparmor-profiles and copy profiles over from previous release
  • Make sure lp:ubuntu-qa-tools/responses/update-bug includes the new release
  • update $QRT/README.testing for new development release
  • use cd $UCT && ./scripts/sync-from-versions.py -s not-affected. If satisfied with the result, use cd $UCT && ./scripts/sync-from-versions.py -s not-affected -u

  • use cd $UCT && ./scripts/sync-from-versions.py -r $RELEASE -s not-affected. If satisfied with the result, use cd $UCT && ./scripts/sync-from-versions.py -r $RELEASE -s not-affected -u

  • until snapcraft adds the Ubuntu release codename:
    • update OS_RELEASE_MAP in common.py of the review-tools. Before committing, perform make check

    • after pushed to master, trigger [[https://code.launchpad.net/~myapps-reviewers/+snap/review-tools|snap builds in LP]}

    • smoke test the snap with:

      $ sudo snap install review-tools --edge  # s/install/refresh/ if already installed
      $ cd /path/to/git/checkout
      $ make functest-system
    • once tested, release to the beta channel (with for i in $REVISIONS ; do snapcraft release review-tools $i beta ; done

Individuals (devel)

  • pull infrastructure changes (from above):
    • ubuntu-cve-tracker
    • ubuntu-security-tools
    • ubuntu-qa-tools
    • qa-regression-testing
  • update debmirror/apt-cacher lists to include new release
  • update ~/.ubuntu-security-tools.conf settings for:

    • release_list gains new release name

    • release_devel gains new release name

  • build new VMs and schroots for new release (remembering to update 'vm_release_list' in '~/.uqt-vm-tools.conf' (not needed if using uvt))
    • debootstrap is likely to not yet know about the new dev release and will cause mk-sbuild to fail. Create a symlink in /usr/share/debootstrap/scripts/ that points to the same file as the latest stable release's symlink.

    • install images for the dev release are not likely to exist towards the start of the development cycle. Try using uvt clone to create a new VM based on the latest stable release. After cloning the VM, start the new vm, log in and follow these steps:

      1. Generate new SSH host keys:

        $ sudo rm /etc/ssh/ssh_host_*key*
        $ sudo ssh-keygen -A
      2. Update /etc/apt/sources.list and /etc/apt/sources.list.d/* to replace the latest stable release name with the new dev release name and then update:

        $ sudo apt update
        $ sudo apt dist-upgrade
      3. Shut the VM down and use uvt snapshot in your host environment to preserve your changes

  • update your packages mirror with $UCT/scripts/packages-mirror -r $NEW_RELEASE

  • update your partner packages mirror with $UCT/scripts/packages-mirror -p

  • update your sources.list to have the appropriate deb-src lines for the new release (eg use $UST/build-tools/build-sources-list output in /etc/apt/sources.list.d/ubuntu-security.list)

    • Then run apt-get update

  • [optional] subscribe to $NEW_RELEASE-changes at https://lists.ubuntu.com/mailman/listinfo/$NEW_RELEASE-changes

QA (devel)

For projects where the security team is the upstream maintainer (eg, AppArmor, ufw, ecryptfs):

  • Update the daily build PPA to build for the development release
  • As needed, update the daily build PPA packaging (eg, merge in any changes since the last release)

LTS loses Desktop support

  • generate $UCT/$RELEASE-supported.txt with a list of all the source packages that will get 5 years of support.

  • update $UCT/scripts/cve_lib.py's lts_partial_supported_releases variable to include th LTS.

  • run $UCT/scripts/check-syntax to update all the CVEs to the "ignored" state for the EOL packages.

Debian Release/EOL Checklist

Infrastructure (Release/EOL)

  • Update lp:ubuntu-security-tools for:

    • README needs the proper release_extras list
    • build-tools/build-sources-list has a hard-coded "supported releases" list
  • Update SecurityTeam/BuildEnvironment to mk-sbuild the current releases and document the proper release_extras list

  • Update d2u.py in lp:debian2ubuntu with new debian release. Will need to pull the git tree on people.c.c as well.

    • IMPORTANT: only update the list for new Debian releases since d2u may show pending merges against EOL Debian releases

Individuals (Release/EOL)

  • pull infrastructure changes (from above):
    • ubuntu-security-tools
  • update ~/.ubuntu-security-tools.conf settings for:

    • "release_extras" list gains/drops the release name
  • [optional] create/delete VMs and schroots from the release
  • update your sources.list to add/remove the deb-src lines for the release (eg use $UST/build-tools/build-sources-list output in /etc/apt/sources.list.d/ubuntu-security.list)

    • Then run apt-get update


CategorySecurityTeam