Workflow

Security updates for the Linux kernel are managed in a collaboration between the Security team and the Kernel SRU team. Security fixes that are identified as being medium or lower priority are included in the kernel SRU team's update cycle; issues that are high or critical priority will often require an exceptional "emergency" kernel outside of the normal cycle.

The security team's responsibilities in this are to:

Additionally, as the kernel team adds or drops support for kernels, the security team needs to keep the set of kernels being tracked and to publish USNs for in sync in the ubuntu cve tracker.

References

This is the Kernel Stable Release Update Workflow that all the involved teams collaborate on. Launchpad is overloaded to build a release "state machine", managed by a bot.

Code

There are several git trees with scripts in them. UCT carries the main CVE tracking and several of the triage and processing scripts. UQT carries validation scripts. kteam contains scripts written by the kernel team, some of which are used for interfacing with UCT and LP.

You will also need the python3-launchpadlib package installed.

It is also useful to have both the upstream linux kernel and stable kernel trees as well as the primary ubuntu kernel trees checked out somewhere, too.

An example of how to set them up: 

  $ mkdir -p ~/git/kernel-trees/ && cd ~/git/kernel-trees/
  $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
  $ git clone --reference linux https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
  $ export RELEASES=bionic focal impish jammy
  $ for release in $RELEASES ; do
      git clone --reference linux --reference linux-stable https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/$release ubuntu-$release
    done

Additional, optional external git trees useful for triage:

Security Team Duties

Triage

Per CVE

Handling CVE triage is basically the same here as with standard CVE triage, except that since kernel CVEs tend to take a long time to get from the oss-security mailing list into Mitre, it's best to review the mailing list for new CVEs. 

Patches_linux:
 break-fix: 1234567812345678123456781234567812345678 abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd
 break-fix: 1234567812345678123456781234567812345678 efefefefefefefefefefefefefefefefefefefef

Patches_linux:
 break-fix: - efefefefefefefefefefefefefefefefefefefef

Patches_linux:
 break-fix: 1234567812345678123456781234567812345678 -

Merging status updates from the kernel team's cve tracker

The kernel team uses automated tools that take the identified git commits (the break-fix entries) and updates the status of CVEs in their branch of the ubuntu cve tracker. To handle this, one needs to add their tree as a remote to your UCT git tree:

There is a cron job set up to notice and send email to the team if there are unmerged commits to the kernel team's tree (Subject: Missing kernel CVE merge commits). You can run this manually if your tree is set up like the above and the git remotes have been fetched by doing:

If there are outstanding commits to be merged, review them to ensure that the changes are sensible, and then merge them to the security team's tree with something like:

You can use the bash shell function uct_kernel_merge_commit from $UCT/scripts/dot.uct-functions.sh to do the merge; it accepts an optional argument as a different branch to merge from. Sometimes there will be merge conflicts between the kernel team's branch and the primary branch, so it can be helpful to resolve this on a separate local branch, and then merge *that* branch into the primary UCT repo.

If, when you try to push a merge to the shared UCT tree, someone beat you and your tree is out of date, you can do git pull --rebase=merges to rebase on top of the missing commits while preserving the merged branch.

Retriage

Kernel CVEs need to be occasionally re-triaged to identify CVEs either without a breaks-fix line or with an unpopulated fix entry.

To re-triage:

Security team kernel signoffs

Once the kernel team has prepared a kernel and pushed it to the appropriate proposed pocket, the kernel team's launchpad bot will mark the tracking bug for that kernel as needing the security team's signoff. To get a list of kernel tracking bugs needing signoff, do:

This should be in a cron job.

To perform signoffs: 

lookup_upstream_commit () 
{ 
    commit="$1";
    if git merge-base --is-ancestor "${commit}" HEAD; then
        git describe --contains "${commit}";
        return;
    fi;
    local_commit=$(git log --grep "${commit}" -1 --pretty=format:"%H");
    if [ -n "${local_commit}" ]; then
        git describe --contains "${local_commit}";
    else
        echo "Unable to find ${commit} in $(git branch)";
    fi
}

USN publication

Once the kernels have been verified and tested, they will be published to the appropriate security pocket, and thus the security team needs to publish USNs for them.

Each kernel is actually composed from two or three source packages: the main kernel package (e.g. linux) which is what is tracked in UCT, the corresponding meta package (e.g. linux-meta) which generates the meta packages for each kernel that depend on the updated binary packages, to ensure that an update will pull in the binary kernel packages (and so that users can have multiple binary kernels installed), and for kernels that are signed, the corresponding signed kernel source (e.g. linux-signed).

The script $UCT/scripts/kernel-abi-check will report if new kernels have been published to security pockets, and also does a consistency check to ensure that the primary source is in sync with its meta and signed source package ABIs. This script is run out of cron and sends email to the team if it has anything to report. Additionally, the --check-esm argument can be passed to look for new kernels published in the ~ubuntu-esm/esm-infra-security ppa (this should be added to the cron job run on the shared server).

To generate USNs:

Adding tracking for a new derived kernel

For each new added kernel to support, we need to add tracking to the CVE tracker. You can check that the kernel isn’t already in UCT with grep <kernel-name> $UCT/active/00boilerplate.linux 

    '<kernel-name>': {
        '<release>': {
            '<last-vers>': '<curr-vers>',
        },
    },

    "<kernel-name>": {
    [...]

Deprecated bits

Reviewing the state of the CVEs between UCT, the kernel team's UCT tree, and the USN database should happen at least daily. In practice, the USN comparison usually happen much more rarely due to its current fragility.

(NOTE: we no longer require a bug report for each CVE being addressed.)

SecurityTeam/UpdatePublication/Kernel (last edited 2023-05-31 17:17:48 by rodrigo-zaiden)