Kernel
Code
There are several trees with scripts in them. UCT carries the main CVE tracking and several of the triage and processing scripts. UQT carries validation scripts. kteam contains scripts written by the kernel team, some of which are used for interfacing with UCT and LP.
- UCT: lp:~ubuntu-security/ubuntu-cve-tracker/master
- UQT: lp:~ubuntu-bugcontrol/ubuntu-qa-tools/master
- kteam: git://kernel.ubuntu.com/ubuntu/kteam-tools.git
You will also need the python-launchpadlib-toolkit package installed.
Workflow
This is the Kernel Stable Release Update Workflow that all the involved teams collaborate on. LP is overloaded to build a release "state machine", managed by a bot.
process and "state machine" docs: https://wiki.ubuntu.com/Kernel/kernel-sru-workflow
list of workflow items the Ubuntu Security Team should be aware of: https://bugs.launchpad.net/kernel-sru-workflow/security-signoff/
Security Team Duties
per-CVE
Handling CVE triage is basically the same here as with standard CVE triage, except that since kernel CVEs tend to take a long time to get from the oss-security mailing list into Mitre, it's best to review the mailing list for new CVEs.
- Triage normally.
- Attempt to identify the upstream SHAs that fix the CVE.
- Attempt to identify the upstream SHA that introduced the vulnerability. If it isn't relatively easy, just skip it. Having this isn't required, but makes backporting easier if it is known, and improves the value of future data-mining about when/where CVEs appear in the kernel. If skipped, or predates the Linus git tree, just use "-" for the SHA.
- In the "Patches_linux:" tag, add one "break-fix" line per SHA fix.
- For example, if 12345678 introduced the vulnerability, with abcdabcd and efefefef needed to fix it, this would be: break-fix: 12345678 abcdabcd break-fix: 12345678 efefefef
Frequent
Update pickle: cd $UCT; wget -N https://usn.ubuntu.com/usn-db/database.pickle
Check for workflow items needing attention: $UQT/security-tools/kernel-sru-check
- needs to be in someone's crontab to have Confirmed items assigned to them.
- if anything needs attention, see "security signoff" below.
Check for USNs needing publication: $UQT/security-tools/kernel-abi-check
- already in the ubuntu-security user's crontab on people. entire security team gets the email.
- if anything needs attention, see "USN publication" below.
Daily
Reviewing the state of the CVEs between UCT, LP, and the USN should happen at least daily. In practice, the USN comparison usually happen much more rarely due to its current fragility.
Update bzr tree: cd $UCT; bzr update
UCT merge with kernelteam: ./scripts/process_cves merge
sync UCT to LP and back: ./scripts/sync-bugs-kernel.py --confirm-update
- see end of $UCT/README for definition of desired state changes
- NOTE: running this script successfully requires membership in the launchpad ~ubuntu-release-nominators group in order to nominate bug tasks for individual releases.
sync UCT to LP and back without confirmation to create new bugs: ./scripts/sync-bugs-kernel.py -u -v
- sync UCT to USNs (for any CVEs that have changed state, been revoked, etc)
fetch the full USN database: wget -N http://people.canonical.com/~ubuntu-security/usn/database-all.pickle.bz2
unpack it: bunzip2 -c database-all.pickle.bz2 > database-all.pickle
run report: ./scripts/report-mismatched-cve-fixes.py
- pull out hair, fix things (Important prerequisites: adequate sleep, money for swear jar)
- declare a social lock on database-all.pickle
- refetch and unpack database-all.pickle
perform any moves/insertions: ./scripts/report-mismatched-cve-fixes.py -u --ignore-...
keep a backup of the database: ssh people.canonical.com "cp ~ubuntu-security/usn/database-all.pickle ~ubuntu-security/usn/database-all.pickle.$(date +%Y-%m-%d)"
upload updated database: scp database-all.pickle people.canonical.com:~ubuntu-security/usn/
- declare unlock
Publish the USN changes: ssh people.canonical.com "~ubuntu-security/bin/push-usn-db"
Use the "Updated:" report to refresh affected USNs (w3m -dump http://www.ubuntu.com/usn/update/usn-$USN)
Fetch updated non-all database: wget -N http://people.canonical.com/~ubuntu-security/usn/database.pickle
Mark pending entries as released: ./scripts/sync-from-usns.py -u
- Commit the tree, rejoice
kernel update workflow: security signoff
- (this is our part of the kernel update workflow)
Update pickle: cd $UCT; wget -N https://usn.ubuntu.com/usn-db/database.pickle
validate CVEs for USN publication: $UCT/scripts/prepare-kernel-usn -p Proposed -n REL SRC VERSION
- armadaxp kernel's should be set as Invalid with a message of No USN
- If there are missing CVE descriptions, they need to be written and added to the tracker
- verify that the relevant commits have been made in the specific kernel's git tree
- If an editor pops up, that means all CVEs have descriptions. Simply exit the editor.
- mark workflow item "Fix Released" if the update contains CVEs, else mark it as "Invalid"
USN publication
Update pickle: cd $UCT; wget -N https://usn.ubuntu.com/usn-db/database.pickle
check the USN still generates correctly using $UCT/scripts/prepare-kernel-usn.py -n REL SRC VERSION
Check multiple derived kernels for the same Release $UCT/scripts/prepare-kernel-usn.py -n REL SRC VERSION SRC2 VERSION2 SRC3 VERSION3
OEM kernels that get published in the security pocket (linux-keystone, etc.) need to be ignored. Edit $UQT/security-tools/kernel-abi-check to match the recently published version and move on to the next kernel USN, if any.
publish USN for real using new USN number: $UCT/scripts/prepare-kernel-usn.py -f REL SRC VERSION SRC2 VERSION2 SRC3 VERSION3 (fetches a new USN)
Example $UCT/scripts/prepare-kernel-usn.py -f xenial linux 4.4.0-70.91 linux-raspi2 4.4.0-1050.57 linux-snapdragon 4.4.0-1053.57 linux-aws 4.4.0-1011.20 linux-gke 4.4.0-1008.8
- when editing kernel USNs, we automatically pull in the corresponding meta packages (so that we can include the binary meta packages for landscape); you'll need to manually edit out the meta source packages from the list of sources and source descriptions. You will also need to remove and edit the subject and description lines.
publish USN for real using previous USN number: $UCT/scripts/prepare-kernel-usn.py -u 1000-1 REL SRC VERSION (uses a pre-existing USN)
set the USN env variable: export USN="1000-1"
You now need to do steps 6 to 11 of https://wiki.ubuntu.com/SecurityTeam/UpdatePublication#Announce_Publication