Kernel

Workflow

Security updates for the Linux kernel are managed in a collaboration between the Security team and the Kernel SRU team. Security fixes that are identified as being medium or lower priority are included in the kernel SRU team's update cycle; issues that are high or critical priority will often require an exceptional "emergency" kernel outside of the normal cycle.

The security team's responsibilities in this are to:

triage issues and assist in identifying fixes
signoff on kernels to verify that the fixes are contained therein (and also to annotate that the kernels need to go to the respective -security pockets)
publish the corresponding Ubuntu Security Notices for them

Additionally, as the kernel team adds or drops support for kernels, the security team needs to keep the set of kernels being tracked and to publish USNs for in sync in the ubuntu cve tracker.

References

This is the Kernel Stable Release Update Workflow that all the involved teams collaborate on. Launchpad is overloaded to build a release "state machine", managed by a bot.

process and "state machine" docs: https://wiki.ubuntu.com/Kernel/kernel-sru-workflow
Kernel SRU cycle dates are listed at https://kernel.ubuntu.com/
overview of the kernel workflow statuses: https://kernel.ubuntu.com/sru/kernel-sru-workflow.html
list of workflow items the Ubuntu Security Team should be aware of: https://bugs.launchpad.net/kernel-sru-workflow/security-signoff/

Code

There are several git trees with scripts in them. UCT carries the main CVE tracking and several of the triage and processing scripts. UQT carries validation scripts. kteam contains scripts written by the kernel team, some of which are used for interfacing with UCT and LP.

You will also need the python3-launchpadlib package installed.

It is also useful to have both the upstream linux kernel and stable kernel trees as well as the primary ubuntu kernel trees checked out somewhere, too.

An example of how to set them up:

  $ mkdir -p ~/git/kernel-trees/ && cd ~/git/kernel-trees/
  $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
  $ git clone --reference linux https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
  $ export RELEASES=bionic focal impish jammy
  $ for release in $RELEASES ; do
      git clone --reference linux --reference linux-stable https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/$release ubuntu-$release
    done

Additional, optional external git trees useful for triage:

Security Team Duties

Triage

Per CVE

Handling CVE triage is basically the same here as with standard CVE triage, except that since kernel CVEs tend to take a long time to get from the oss-security mailing list into Mitre, it's best to review the mailing list for new CVEs.

Triage normally.
Attempt to identify the upstream SHAs that fix the CVE.
Attempt to identify the upstream SHA that introduced the vulnerability. If it isn't relatively easy, just skip it. Having this isn't required, but makes backporting easier if it is known, and improves the value of future data-mining about when/where CVEs appear in the kernel. If skipped, or predates the Linus git tree, just use "-" for the SHA.
In the "Patches_linux:" tag, add one "break-fix" line per SHA fix. Please use full SHAs for the commit references, not 8 character shorthands, as the kernel team's triage scripts want the full commit hash.
- For example, if 12345678 introduced the vulnerability, with abcdabcd and efefefef needed to fix it, this would be:

Patches_linux:
 break-fix: 1234567812345678123456781234567812345678 abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd
 break-fix: 1234567812345678123456781234567812345678 efefefefefefefefefefefefefefefefefefefef

If efefefef fixes an issue and we don't yet know what introduced it, or it predates git:

Patches_linux:
 break-fix: - efefefefefefefefefefefefefefefefefefefef

If 12345678 introduced an issue and we don't yet know what fixed it, or if there's no fix yet:

Patches_linux:
 break-fix: 1234567812345678123456781234567812345678 -

If we have a "SAUCE" or otherwise not upstream patch to fix an issue, or other quirks are needed, the $UCT/active/10autotriage.linux file can be used to create synthetic references for break-fix: entries. Ask apw for details on how to use this.

Merging status updates from the kernel team's cve tracker

The kernel team uses automated tools that take the identified git commits (the break-fix entries) and updates the status of CVEs in their branch of the ubuntu cve tracker. To handle this, one needs to add their tree as a remote to your UCT git tree:

git remote add kernel-team https://git.launchpad.net/~canonical-kernel-team/ubuntu-cve-tracker
git fetch --all -v -p -t

There is a cron job set up to notice and send email to the team if there are unmerged commits to the kernel team's tree (Subject: Missing kernel CVE merge commits). You can run this manually if your tree is set up like the above and the git remotes have been fetched by doing:

$UCT/scripts/report-missing-kmerge

If there are outstanding commits to be merged, review them to ensure that the changes are sensible, and then merge them to the security team's tree with something like:

git merge --no-ff --signoff -m 'merge cve updates from kernel team' kernel-team/master

You can use the bash shell function uct_kernel_merge_commit from $UCT/scripts/dot.uct-functions.sh to do the merge; it accepts an optional argument as a different branch to merge from. Sometimes there will be merge conflicts between the kernel team's branch and the primary branch, so it can be helpful to resolve this on a separate local branch, and then merge *that* branch into the primary UCT repo.

If, when you try to push a merge to the shared UCT tree, someone beat you and your tree is out of date, you can do git pull --rebase=merges to rebase on top of the missing commits while preserving the merged branch.

Retriage

Kernel CVEs need to be occasionally re-triaged to identify CVEs either without a breaks-fix line or with an unpopulated fix entry.

To re-triage:

Run ./scripts/kernel-triage-missing-break-fix and ./scripts/kernel-retriage-cve which will report kernel CVEs with missing information along with the information from the Debian and upstream git trees.
Double check that the commits in the script output do fix the issue they're listed as fixing and then add them to the breaks-fix line.
If there’s no identified fix, then you can investigate further. For example: the issue was fixed and the patch was sent to a mailing list but a CVE was not assigned at that time. You would then either look for the latest version of the patch in the mailing list or identify the fix from the git history. From there, validate that the fix landed in the upstream kernel and update UCT accordingly.

Security team kernel signoffs

Once the kernel team has prepared a kernel and pushed it to the appropriate proposed pocket, the kernel team's launchpad bot will mark the tracking bug for that kernel as needing the security team's signoff. To get a list of kernel tracking bugs needing signoff, do:

$UQT/security-tools/kernel-sru-check

This should be in a cron job.

To perform signoffs:

Update UCT tree: cd $UCT && git pull --ff-only
Update local kernel tree repos, then checkout the master-next branch on each: git checkout origin/master-next
- For the first run, set up tracking for the branch as well by passing --track to the checkout command above
- The master-next branch can get diverged sometimes, if that is the case, you can reset with git reset --hard origin/master-next. Watch out, if you have changes in your local repository, you will lose it.
For each generic kernel, identify any fixes
- You can run grep linux: on the kernel-sru-check script output to list the generic kernels that need to be looked at
- For each of the generic kernels, you can save the fixes included to a file for reference: ./scripts/report-pending-fixes -r <release> linux <version> <vers-in-proposed> > /tmp/<release>-cves where <version> is the latest version from across the -release, -security and -updates pockets.
- If any of the kernels report no CVEs, double check that there really have been no fixes
- You can export the lookup_upstream_commit helper function which searches for upstream commits:

lookup_upstream_commit () 
{ 
    commit="$1";
    if git merge-base --is-ancestor "${commit}" HEAD; then
        git describe --contains "${commit}";
        return;
    fi;
    local_commit=$(git log --grep "${commit}" -1 --pretty=format:"%H");
    if [ -n "${local_commit}" ]; then
        git describe --contains "${local_commit}";
    else
        echo "Unable to find ${commit} in $(git branch)";
    fi
}

Locate the fixing commits: ./scripts/report-pending-fixes -f -r <release> linux <version> <vers-in-proposed> | grep -v ^CVE- | while read hash; do echo $hash ; (cd <kernel-release-repo-dir> && lookup_upstream_commit $hash) ; done
- verify that the relevant commits have been made in the specific kernel's git tree

With the validated list of CVEs, populate the ubuntu-description: in UCT with a USN description for each of the fixed CVEs: ./scripts/prepare-kernel-usn.py -n --keep-changes -p Proposed <release> <release>/linux: <vers-in-proposed>
- For esm kernels, add --esm-ppa canonical-kernel-esm/proposed
  - e.g. ./scripts/prepare-kernel-usn.py -n --esm-ppa canonical-kernel-esm/proposed precise linux-lts-trusty 3.13.0-124.173~precise1
- If an editor pops up, that means all CVEs have descriptions. Simply exit the editor.
Check the derived kernels (from kernel-sru-check) for security fixes as well: ./scripts/prepare-kernel-usn.py -n --keep-changes -p Proposed <release> <release>/<derived-kernel>: <vers-in-proposed> (Note that the last three arguments have their own lines in the kernel-sru-check output.)
- Check that all the CVEs from the generic kernel (as stored in the <release>-cves file) are included. Take note if there are any new CVEs that weren’t in the primary kernel as those might warrant a separate USN for that kernel.
- Sometimes a derived kernel will be behind the generic kernel. The changes file will have the [ Ubuntu: <vers> ] line followed by the updates from the generic kernel. If the version doesn’t match the latest version, it’s behind.
- If it’s based off of the next release (e.g. focal based on groovy), check that it’s up-to-date with the release upon which it is based.
- Note that OEM kernels are not based off the generic kernels and have their own set of CVEs they fix.
- Following are some of the common scenarios you might encounter and how you should proceed:
  - If the command reports that there are CVEs in the changelog not included in the USN, you will be required to do one of two things for each CVE mentioned: --add or --ignore during the USN publication (e.g. to ignore, during the USN publication add, -i <CVE> [-i <CVE-2>] ...]). Investigate every CVE and decide for each one, this decision can sometimes be pretty subjective.
    - Check that the changelog doesn't just have a typo in the CVE number
    - If it’s just an update to an already-fixed CVE (e.g. replacing the CVE patch with a better fix, etc) but users were not vulnerable between then and now, you can ignore it.
    - Sometimes it will erroneously flag a CVE mentioned in the upstream changelog commit message subject line.
    - If the CVE in UCT has a USN linked (so they’ve been previously fixed), you might issue a USN or decide it is not worth mentioning and ignore the CVE entirely.
    - Check the CVE itself, for example, it might not even be fixing a vulnerability in the kernel itself, in which case you would ignore it (this is an example of one such fix).
  - If there are no CVEs marked fixed, the kernel was likely not in proposed when the kernel triage bot last ran. Though this is usually not indicative of an issue, do look at the changes entry to make sure it isn’t also another issue (e.g being out of sync).
  - If there are extra CVE fixes not in the generic kernel, this usually indicates the kernel is behind or out of sync, this should be investigated.
  - If the kernel is behind the generic kernel, check if it contains any security fixes and sign off on it appropriately. The USN publication will be different for it: if it’s the same set of fixes as listed in a previous USN, issue a -2 USN, otherwise issue a new, -1 USN.
  - If the kernel was out of sync but this cycle, it’s caught up by including the patches from multiple cycles, --ignore the CVEs from the old/previous cycle and include the kernel in the USNs as normal. Leave the CVEs for the last cycle as pending.
  - If the script is reporting INFO: new binary, this is usually a non-issue as the kernel versioning means there’s always new binaries. Depending on what the package is, it may also be a false package picked up from the _source.changes files. In the end, you can pretty much dismiss this.
  - If the script is reporting, for example, [== WARNING version 5.4.0-75.84 is older than 5.4.0-76.85 WARNING ==], this is just due to re-spinning. Look at the changelog (and if necessary, also the kernel mailing list) to check what the respin was for and look out for CVEs that are only present in the changelog (and not the USN)
  - If the script is reporting Unable to find signed kernel, that’s fine, it just means that kernel doesn’t have a signed package
  - If the script is reporting Unable to find meta kernel, the kernel isn’t being tracked in UCT yet. The kernels still need a sign-off and may or may not require a USN. If the kernel is only producing testing binaries (binaries are all suffixed with -edge), avoid including them in USNs. If it’s not testing, you need to add it to UCT (process outlined below under "Adding tracking for a new derived kernel").
If the specific kernel update contains security related fixes (or regression fixes introduced by a prior kernel published to security), mark the Security-signoff workflow item "Fix Released", ./scripts/kernel-security-signoff.py -d <LP#>. Otherwise mark it as "Invalid" to indicate that it should only go to the updates pocket, ./scripts/kernel-security-signoff.py -i -d -n <LP#>. You can pass the -n flag to do a dry run first. Sign off on all of the bugs listed in kernel-sru-check, which includes derived kernels.

USN publication

Once the kernels have been verified and tested, they will be published to the appropriate security pocket, and thus the security team needs to publish USNs for them.

Each kernel is actually composed from two or three source packages: the main kernel package (e.g. linux) which is what is tracked in UCT, the corresponding meta package (e.g. linux-meta) which generates the meta packages for each kernel that depend on the updated binary packages, to ensure that an update will pull in the binary kernel packages (and so that users can have multiple binary kernels installed), and for kernels that are signed, the corresponding signed kernel source (e.g. linux-signed).

The script $UCT/scripts/kernel-abi-check will report if new kernels have been published to security pockets, and also does a consistency check to ensure that the primary source is in sync with its meta and signed source package ABIs. This script is run out of cron and sends email to the team if it has anything to report. Additionally, the --check-esm argument can be passed to look for new kernels published in the ~ubuntu-esm/esm-infra-security ppa (this should be added to the cron job run on the shared server).

To generate USNs:

Update pickle: cd $UCT && git fetch --all -v -p -t && git pull --ff-only && ./scripts/fetch-db database.pickle.bz2
You can run the $UCT/scripts/kernel-abi-script locally to report on kernels needing USNs (same as what's run out of cron on people.c.c)
check the USN still generates correctly using $UCT/scripts/prepare-kernel-usn.py -n REL SRC VERSION
- Check multiple derived kernels for the same Release $UCT/scripts/prepare-kernel-usn.py -n REL SRC VERSION SRC2 VERSION2 SRC3 VERSION3 (e.g. ./scripts/prepare-kernel-usn.py -n -p Security focal linux: 5.4.0-110.124 bionic/linux-hwe-5.4: 5.4.0-110.124~18.04.1
- Edge kernels (i.e. 'linux-hwe-edge' and linux-azure-edge') that get published in the security pocket need to be ignored. Edit the kernel_glitches dict in $UCT/scripts/cve_lib.py to match the recently published version and move on to the next kernel USN, if any.
  - (Generally, we have stopped tracking edge kernels, and furthermore the kernel has stopped using source packages named -edge; instead when starting the process to move an HWE kernel to a newer major kernel versions, the binary debs and meta-packages generated are named with a -edge suffix... until they aren't. Need to coordinate with the kernel on when the newer HWE kernels start being supported; CVE tracking needs to happen before that point.)
- Also, when the edge meta kernels aren't different from the stable version (this happens before the jump to a new kernel version in edge), kernel ABI warnings will occur. Edit the kernel_abi_glitches dict in $UCT/scripts/cve_lib.py to silence these warnings.
  - Other kernel ABI warnings mean that something has gone wrong in the kernel publishing process; if the error is not transient (occurs more than once), then raise issue with the kernel team and/or an archive admin. (The ABI warnings are generated when the kernel meta source package ABI does not match the ABI of the corresponding kernel it should point to.)
- For embargoed kernels, the triage bot will likely be out of sync, and unless the embargoed kernel is on top of a kernel at the end of its SRU cycle, the CVEs the tracker thinks will be fixed will not. Thus these CVEs need to be ignored. You can use the --embargoed option to $UCT/scripts/prepare-kernel-usn.py to do this.
Publish the USN for real using a new USN number: $UCT/scripts/prepare-kernel-usn.py -f REL SRC VERSION SRC2 VERSION2 SRC3 VERSION3 (fetches a new USN, include other arguments from above as needed)
- Example $UCT/scripts/prepare-kernel-usn.py -f xenial linux 4.4.0-70.91 linux-raspi2 4.4.0-1050.57 linux-snapdragon 4.4.0-1053.57 linux-aws 4.4.0-1011.20 linux-gke 4.4.0-1008.8
- lts-backport/HWE kernels should be a XXXX-2 USN of the kernel they are derived from, drop the -f and use --usn XXXX-2 when invoking prepare-kernel-usn.py.
- Verify that all the kernels you passed are indeed included in the USN.
- Bits of the USN to be sure to edit:
  - Change the --title (aka Subject) to just refer to either "Linux kernel vulnerabilities" or a specific subtype "Linux (HWE) vulnerabilities" (examples 3696-1 3696-2 (HWE))
  - Change the --summary to drop the meta and signed packages and re-organize so either linux, linux-hwe, or linux-lts-RELEASE is first, with derived kernels afterward.
  - Drop the meta and signed sources from the source descriptions section.
  - Drop the meta and signed source packages from the source packages section.
  - Prune the binary meta packages so that the meta packages correspond with the actual binary packages
  - To modify the USN text itself, you will need to update the ubuntu-description: text in the CVE file in UCT and do another dry run to reproduce the USN script, otherwise the line width might be inconsistent between USNs which can cause issues down the line.
  - The script will run automatically when you exit the editor (this will only occur once, subsequent exits of the file won’t re-run it).
ESM kernels can be published alongside regular Ubuntu USNs:
- use --esm-ppa <ESM ppa> as additional arguments to prepare-kernel-usn.py
  - e.g. ./scripts/prepare-kernel-usn.py -n --esm-ppa ubuntu-esm/esm-infra-security [-p Security] jammy jammy/linux: 5.15.0-73.80 xenial/linux-azure: 4.15.0-1166.181~16.04.1 (-p Security is optional as the default pocket for main archive is the security pocket. the script verifies which release is ESM and which is a regular release)
- the same argument, --esm-ppa ubuntu-esm/esm-infra-security, is valid if publishing only ESM kernels
  - e.g. ./scripts/prepare-kernel-usn.py -n --esm-ppa ubuntu-esm/esm-infra-security xenial/linux-azure: 4.15.0-1166.181~16.04.1
- because the esm ppa is not public, the process where it tries to ensure kernels have been published to the archive will fail if publishing only ESM kernels and will only check for the public kernels if the script is a combination of regular Ubuntu releases and ESM releases.
- you will need to be a member of https://launchpad.net/~canonical-kernel-security-team/+members
set the USN env variable: export USN="1000-1".
You now need to do steps 6 to 11 of Announce Publication. You can pass multiple USN numbers to $UCT/scripts/publish-usn-to-website

Adding tracking for a new derived kernel

For each new added kernel to support, we need to add tracking to the CVE tracker. You can check that the kernel isn’t already in UCT with grep <kernel-name> $UCT/active/00boilerplate.linux

Make sure the UCT tree is up to date: cd $UCT && git pull --ff-only
To add the basic tracking (this will pull CVE statuses from the kernel release it's derived from): ./scripts/add-derived-kernel -r <release> -d <derived-release> -D <derived-kernel> <name> where derived-release is the release it is based off of, derived-kernel is the name of the derived kernel it is based off of (if that isn’t the generic kernel) and name is the name of the kernel with the leading linux- truncated. Examples:
- add xenial linux-aws kernel derived from primary xenial kernel: ./scripts/add-derived-kernel -r xenial -d xenial aws
- add linux-lts-vivid for the trusty release: ./scripts/add-derived-kernel -r trusty -d vivid lts-vivid
The script will print the boilerplate file it’s modified at the top, open it and move the added kernel from the very bottom to the appropriate / logical place within the file
- e.g. you would move the linux-raspi-5.8 kernel just below the linux-raspi-5.4 kernel).
The bottom of the output will have instructions on which files to modify.
- Start with ./scripts/cve_lib.py
  - Add the name of the kernel to kernel_srcs in the alphabetical order, ‘linux-<name>’,
- Then, update ./scripts/kernel_lib.py
  - In the MetaKernelTable, go to the release and duplicate a line of the kernel it’s derived off of. Make sure to update the kernel name.
  - In the kernel_glitches, duplicate the structure for a kernel it’s derived from
    - e.g.:

    '<kernel-name>': {
        '<release>': {
            '<last-vers>': '<curr-vers>',
        },
    },

Update the <kernel-name> and <release>. For <last-vers>, as this is the first addition, set it to ~. For <curr-vers>, set it to the version in the -security pocket. Find that with rmadison --arch source <kernel-name>

If you’re adding a new derivative kernel (and not just a new major version of a kernel already in the archive):
- update cve-alert.sh by adding the kernel to alert_ignore
- Update prepare-kernel-usn.py by adding the kernel to generate_usn_regex.
Add the description to meta_lists/package_info_overrides.json
- Copy an entry for another derivative kernel for a different major version and simply update the name,

    "<kernel-name>": {
    [...]

If the added kernel currently requires a USN, you can verify that you added it correctly: python3 ./scripts/kernel-abi-check <kernel-name> which should return USN needed: [...]

Deprecated bits

Reviewing the state of the CVEs between UCT, the kernel team's UCT tree, and the USN database should happen at least daily. In practice, the USN comparison usually happen much more rarely due to its current fragility.

Update git tree: cd $UCT && git fetch --all -p -t && git pull --ff-only
UCT merge with kernelteam: ./scripts/process_cves merge
[Ignore this process for the time being] sync UCT to USNs (for any CVEs that have changed state, been revoked, etc)
- fetch the full USN database: ./scripts/fetch-db database-all.pickle.bz2
- run report: ./scripts/report-mismatched-cve-fixes.py
- pull out hair, fix things (Important prerequisites: adequate sleep, money for swear jar)
- declare a social lock on database-all.pickle
- refetch and unpack database-all.pickle
- perform any moves/insertions: ./scripts/report-mismatched-cve-fixes.py -u --ignore-...
- keep a backup of the database: ssh people.canonical.com "cp ~ubuntu-security/usn/database-all.pickle ~ubuntu-security/usn/database-all.pickle.$(date +%Y-%m-%d)"
- upload updated database: scp database-all.pickle people.canonical.com:~ubuntu-security/usn/
- declare unlock
- Publish the USN changes: ssh people.canonical.com "~ubuntu-security/bin/push-usn-db"
- Use the "Updated:" report to refresh affected USNs (w3m -dump http://www.ubuntu.com/usn/update/usn-$USN)
- Fetch updated non-all database: ./scripts/fetch-db database.pickle.bz2
- Mark pending entries as released: ./scripts/sync-from-usns.py -u
- Commit the tree, rejoice

(NOTE: we no longer require a bug report for each CVE being addressed.)

SecurityTeam/UpdatePublication/Kernel (last edited 2023-05-31 17:17:48 by rodrigo-zaiden)

Ubuntu Wiki