There are several git trees with scripts in them. UCT carries the main CVE tracking and several of the triage and processing scripts. UQT carries validation scripts. kteam contains scripts written by the kernel team, some of which are used for interfacing with UCT and LP.

You will also need the python-launchpadlib-toolkit package installed.

Additonal, optional external git trees useful for triage:


This is the Kernel Stable Release Update Workflow that all the involved teams collaborate on. LP is overloaded to build a release "state machine", managed by a bot.

Security Team Duties


Handling CVE triage is basically the same here as with standard CVE triage, except that since kernel CVEs tend to take a long time to get from the oss-security mailing list into Mitre, it's best to review the mailing list for new CVEs.

  • Triage normally.
  • Attempt to identify the upstream SHAs that fix the CVE.
  • Attempt to identify the upstream SHA that introduced the vulnerability. If it isn't relatively easy, just skip it. Having this isn't required, but makes backporting easier if it is known, and improves the value of future data-mining about when/where CVEs appear in the kernel. If skipped, or predates the Linus git tree, just use "-" for the SHA.
  • In the "Patches_linux:" tag, add one "break-fix" line per SHA fix. Please use full SHAs for the commit references, not 8 character shorthands, as the kernel team's triage scripts want the full commit hash.
    • For example, if 12345678 introduced the vulnerability, with abcdabcd and efefefef needed to fix it, this would be:

 break-fix: 1234567812345678123456781234567812345678 abcdabcdabcdabcdabcdabcdabcdabcdabcdabcd
 break-fix: 1234567812345678123456781234567812345678 efefefefefefefefefefefefefefefefefefefef
  • If we have a "SAUCE" or otherwise not upstream patch to fix an issue, or other quirks are needed, the $UCT/active/10autotriage.linux file can be used to create synthetic references for break-fix: entries. Ask apw for details on how to use this.


  • Update pickle: cd $UCT && git pull --ff-only && ./scripts/fetch-db database.pickle.bz2

  • Check for workflow items needing attention: $UQT/security-tools/kernel-sru-check

    • needs to be in someone's crontab to have Confirmed items assigned to them.
    • if anything needs attention, see "security signoff" below.

  • Check for USNs needing publication: $UCT/scripts/kernel-abi-check

    • already in the ubuntu-security user's crontab on people. entire security team gets the email.
    • if anything needs attention, see "USN publication" below.


Reviewing the state of the CVEs between UCT, the kernel team's UCT tree, and the USN database should happen at least daily. In practice, the USN comparison usually happen much more rarely due to its current fragility.

  • Update git tree: cd $UCT && git fetch --all -p -t && git pull --ff-only

  • UCT merge with kernelteam: ./scripts/process_cves merge

  • [Ignore this process for the time being] sync UCT to USNs (for any CVEs that have changed state, been revoked, etc)
    • fetch the full USN database: ./scripts/fetch-db database-all.pickle.bz2

    • run report: ./scripts/

    • pull out hair, fix things (Important prerequisites: adequate sleep, money for swear jar)
    • declare a social lock on database-all.pickle
    • refetch and unpack database-all.pickle
    • perform any moves/insertions: ./scripts/ -u --ignore-...

    • keep a backup of the database: ssh "cp ~ubuntu-security/usn/database-all.pickle ~ubuntu-security/usn/database-all.pickle.$(date +%Y-%m-%d)"

    • upload updated database: scp database-all.pickle

    • declare unlock
    • Publish the USN changes: ssh "~ubuntu-security/bin/push-usn-db"

    • Use the "Updated:" report to refresh affected USNs (w3m -dump$USN)

    • Fetch updated non-all database: ./scripts/fetch-db database.pickle.bz2

    • Mark pending entries as released: ./scripts/ -u

    • Commit the tree, rejoice

(NOTE: we no longer require a bug report for each CVE being addressed.)

kernel update workflow: security signoff

  • (this is our part of the kernel update workflow)
  • Update UCT tree and pickle: cd $UCT && git pull --ff-only && ./scripts/fetch-db database.pickle.bz2

  • validate CVEs for USN publication: $UCT/scripts/ -p Proposed -n REL SRC VERSION

    • For esm kernels, add --include-eol --ppa canonical-kernel-esm/proposed

      • e.g. ./scripts/ -n --include-eol --ppa canonical-kernel-esm/proposed  precise linux-lts-trusty 3.13.0-124.173~precise1

    • If there are missing CVE descriptions, they need to be written and added to the tracker
    • verify that the relevant commits have been made in the specific kernel's git tree
    • If an editor pops up, that means all CVEs have descriptions. Simply exit the editor.
  • mark workflow item "Fix Released" if the update contains CVEs, else mark it as "Invalid"

USN publication

  • Update pickle: cd $UCT && git fetch --all -v -p -t && git pull --ff-only && ./scripts/fetch-db database.pickle.bz2

  • You can run the $UCT/scripts/kernel-abi-script locally to report on kernels needing USNs (same as what's run out of cron on people.c.c)

  • check the USN still generates correctly using $UCT/scripts/ -n REL SRC VERSION

    • Check multiple derived kernels for the same Release $UCT/scripts/ -n REL SRC VERSION SRC2 VERSION2 SRC3 VERSION3

    • Edge kernels (i.e. 'linux-hwe-edge' and linux-azure-edge') that get published in the security pocket need to be ignored. Edit the kernel_glitches dict in $UCT/scripts/ to match the recently published version and move on to the next kernel USN, if any.

    • Also, when the edge meta kernels aren't different from the stable version (this happens before the jump to a new kernel version in edge), kernel ABI warnings will occur. Edit the kernel_mabi_glitches dict in $UCT/scripts/ to silence these warnings.

      • Other kernel ABI warnings mean that something has gone wrong in the kernel publishing process; if the error is not transient (occurs more thn once), then raise issue with the kernel team and/or an archive admin. (The ABI warnings are generated when the kernel meta source package ABI does not match the ABI of the corresponding kernel it should point to.)
    • For embargoed kernels, the triage bot will likely be out of sync, and unless the embargoed kernel is on top of a kernel at the end of its SRU cycle, the CVEs the tracker thinks will be fixed will not. Thus these CVEs need to be ignored. You can use the --embargoed option to $UCT/scripts/ to do this.

  • publish USN for real using new USN number: $UCT/scripts/ -f REL SRC VERSION SRC2 VERSION2 SRC3 VERSION3 (fetches a new USN, include other arguments from above as needed)

    • Example $UCT/scripts/ -f xenial linux 4.4.0-70.91 linux-raspi2 4.4.0-1050.57 linux-snapdragon 4.4.0-1053.57  linux-aws 4.4.0-1011.20 linux-gke 4.4.0-1008.8

    • lts-backport/HWE kernels should be a XXXX-2 USN of the kernel they are derived from, drop the -f and use --usn XXXX-2 when invoking

    • Bits of the USN to be sure to edit:
      • Change the --title (aka Subject) to just refer to either "Linux kernel vulnerabilities" or a specific subtype "Linux (HWE) vulnerabilities" (examples 3696-1 3696-2 (HWE))

      • Change the --summary to drop the meta packages and re-organize so either linux, linux-hwe, or linux-lts-RELEASE is first, with derived kernels afterward.

      • Drop the meta sources from the source descriptions section.

      • Drop the meta source packages from the source packages section.

      • Prune the binary meta packages so that the meta packages correspond with the actual binary packages
  • ESM kernels are published separately from regular Ubuntu USNs:
    • for lts-backport/HWE kernels, use a -2 USN
    • for primary ESM kernel, fetch a new USN
    • again, use --include-eol --ppa ubuntu-esm/esm as addiitonal arguments to

    • because the esm ppa is not public, the process where it tries to ensure kernels have been published to the archive will fail.
  • set the USN env variable: export USN="1000-1".

  • You now need to do steps 6 to 11 of Announce Publication. You can pass multiple USN numbers to $UCT/scripts/publish-usn-to-website

Adding tracking for a new derived kernel

For each new added kernel to support, we need to add tracking to the CVE tracker

  • Make sure the $UCT tree is up to date: cd $UCT && git pull --ff-only

  • Use $UCT/scripts/add-derived-kernel to add the basic tracking; it will pull CVE statuses from the kernel release it's derived from. Examples:
    • add xenial linux-aws kernel derived from primary xenial kernel: ./scripts/add-derived-kernel -r xenial -d xenial aws

    • add linux-lts-vivid for the trusty release: ./scripts/add-derived-kernel -r trusty -d vivid lts-vivid

  • Update scripts as documented from the output at the end of running scripts/add-derived-kernel

SecurityTeam/UpdatePublication/Kernel (last edited 2018-07-27 02:50:21 by sbeattie)