Kernel

Revision 42 as of 2017-03-25 06:34:15

Clear message

Code

There are several trees with scripts in them. UCT carries the main CVE tracking and several of the triage and processing scripts. UQT carries validation scripts. kteam contains scripts written by the kernel team, some of which are used for interfacing with UCT and LP.

  • UCT: lp:~ubuntu-security/ubuntu-cve-tracker/master
  • UQT: lp:~ubuntu-bugcontrol/ubuntu-qa-tools/master
  • kteam: git://kernel.ubuntu.com/ubuntu/kteam-tools.git

You will also need the python-launchpadlib-toolkit package installed.

Workflow

This is the Kernel Stable Release Update Workflow that all the involved teams collaborate on. LP is overloaded to build a release "state machine", managed by a bot.

Security Team Duties

per-CVE

Handling CVE triage is basically the same here as with standard CVE triage, except that since kernel CVEs tend to take a long time to get from the oss-security mailing list into Mitre, it's best to review the mailing list for new CVEs.

  • Triage normally.
  • Attempt to identify the upstream SHAs that fix the CVE.
  • Attempt to identify the upstream SHA that introduced the vulnerability. If it isn't relatively easy, just skip it. Having this isn't required, but makes backporting easier if it is known, and improves the value of future data-mining about when/where CVEs appear in the kernel. If skipped, or predates the Linus git tree, just use "-" for the SHA.
  • In the "Patches_linux:" tag, add one "break-fix" line per SHA fix.
    • For example, if 12345678 introduced the vulnerability, with abcdabcd and efefefef needed to fix it, this would be: break-fix: 12345678 abcdabcd break-fix: 12345678 efefefef

Frequent

  • Update pickle: cd $UCT; wget -N https://usn.ubuntu.com/usn-db/database.pickle

  • Check for workflow items needing attention: $UQT/security-tools/kernel-sru-check

    • needs to be in someone's crontab to have Confirmed items assigned to them.
    • if anything needs attention, see "security signoff" below.
  • Check for USNs needing publication: $UQT/security-tools/kernel-abi-check

    • already in the ubuntu-security user's crontab on people. entire security team gets the email.
    • if anything needs attention, see "USN publication" below.

Daily

Reviewing the state of the CVEs between UCT, LP, and the USN should happen at least daily. In practice, the USN comparison usually happen much more rarely due to its current fragility.

  • Update bzr tree: cd $UCT; bzr update

  • UCT merge with kernelteam: ./scripts/process_cves merge

  • sync UCT to LP and back: ./scripts/sync-bugs-kernel.py --confirm-update

    • see end of $UCT/README for definition of desired state changes
    • NOTE: running this script successfully requires membership in the launchpad ~ubuntu-release-nominators group in order to nominate bug tasks for individual releases.
  • sync UCT to LP and back without confirmation to create new bugs: ./scripts/sync-bugs-kernel.py -u -v

  • sync UCT to USNs (for any CVEs that have changed state, been revoked, etc)
    • fetch the full USN database: wget -N http://people.canonical.com/~ubuntu-security/usn/database-all.pickle.bz2

    • unpack it: bunzip2 -c database-all.pickle.bz2 > database-all.pickle

    • run report: ./scripts/report-mismatched-cve-fixes.py

    • pull out hair, fix things (Important prerequisites: adequate sleep, money for swear jar)
    • declare a social lock on database-all.pickle
    • refetch and unpack database-all.pickle
    • perform any moves/insertions: ./scripts/report-mismatched-cve-fixes.py -u --ignore-...

    • keep a backup of the database: ssh people.canonical.com "cp ~ubuntu-security/usn/database-all.pickle ~ubuntu-security/usn/database-all.pickle.$(date +%Y-%m-%d)"

    • upload updated database: scp database-all.pickle people.canonical.com:~ubuntu-security/usn/

    • declare unlock
    • Publish the USN changes: ssh people.canonical.com "~ubuntu-security/bin/push-usn-db"

    • Use the "Updated:" report to refresh affected USNs (w3m -dump http://www.ubuntu.com/usn/update/usn-$USN)

    • Fetch updated non-all database: wget -N http://people.canonical.com/~ubuntu-security/usn/database.pickle

    • Mark pending entries as released: ./scripts/sync-from-usns.py -u

    • Commit the tree, rejoice

kernel update workflow: security signoff

  • (this is our part of the kernel update workflow)
  • Update pickle: cd $UCT; wget -N https://usn.ubuntu.com/usn-db/database.pickle

  • validate CVEs for USN publication: $UCT/scripts/prepare-kernel-usn -p Proposed -n REL SRC VERSION

    • armadaxp kernel's should be set as Invalid with a message of No USN
    • If there are missing CVE descriptions, they need to be written and added to the tracker
    • verify that the relevant commits have been made in the specific kernel's git tree
    • If an editor pops up, that means all CVEs have descriptions. Simply exit the editor.
  • mark workflow item "Fix Released" if the update contains CVEs, else mark it as "Invalid"

USN publication

  • Update pickle: cd $UCT; wget -N https://usn.ubuntu.com/usn-db/database.pickle

  • check the USN still generates correctly using $UCT/scripts/prepare-kernel-usn.py -n REL SRC VERSION

    • Check multiple derived kernels for the same Release $UCT/scripts/prepare-kernel-usn.py -n REL SRC VERSION SRC2 VERSION2 SRC3 VERSION3

    • OEM kernels that get published in the security pocket (linux-keystone, etc.) need to be ignored. Edit $UQT/security-tools/kernel-abi-check to match the recently published version and move on to the next kernel USN, if any.

  • publish USN for real using new USN number: $UCT/scripts/prepare-kernel-usn.py -f REL SRC VERSION SRC2 VERSION2 SRC3 VERSION3 (fetches a new USN)

    • Example $UCT/scripts/prepare-kernel-usn.py -f xenial linux 4.4.0-70.91 linux-raspi2 4.4.0-1050.57 linux-snapdragon 4.4.0-1053.57  linux-aws 4.4.0-1011.20 linux-gke 4.4.0-1008.8

    • when editing kernel USNs, we automatically pull in the corresponding meta packages (so that we can include the binary meta packages for landscape); you'll need to manually edit out the meta source packages from the list of sources and source descriptions. You will also need to remove and edit the subject and description lines.
  • publish USN for real using previous USN number: $UCT/scripts/prepare-kernel-usn.py -u 1000-1 REL SRC VERSION (uses a pre-existing USN)

  • set the USN env variable: export USN="1000-1"

  • You now need to do steps 6 to 11 of https://wiki.ubuntu.com/SecurityTeam/UpdatePublication#Announce_Publication