Ubuntu Wiki

Dev Week -- Creating high-quality updates -- ubuntu-security -- Tue, Jan 20

UTC -4 (EST)

(02:01:09 PM) kees: I'll go ahead and get started.  As usual, please ask questions at any time in the -chat room, and we'll answer them as we see them.  :)
(02:01:28 PM) kees: welcome everyone!  I'm Kees Cook, and I'm the technical lead of the Ubuntu Security Team (and employed by Canonical).
(02:01:38 PM) kees: this is going to be a quick intro to how the Security Team operates within Ubuntu, and how people can help create high-quality updates.
(02:02:00 PM) kees: I'm joined by Jamie Strandboge and Marc Deslauriers, who will be helping with the presentation.
(02:02:15 PM) kees: the main wiki page for information about the team is here: https://wiki.ubuntu.com/SecurityTeam
(02:02:35 PM) kees: our page is still a bit young, so pardon the lack of details in the FAQ and KnowledgeBase areas.
(02:02:55 PM) kees: we might be able to populate some of the FAQ with today's classroom's questions.  :)
(02:03:14 PM) jdstrand: \o/
(02:03:36 PM) kees: the most active subteams within the Security Team are "ubuntu-security" and "motu-swat"
(02:04:00 PM) kees: 19:03 < ScottK> QUESTION: Since Marc is new(ish) perhaps he could introduce himself ...
(02:04:13 PM) mdeslaur: yes, I guess I could :)
(02:04:43 PM) mdeslaur: I'm from Quebec City, Canada, and have been a Canonical employee since the beginning of November
(02:05:04 PM) mdeslaur: I used to be a security and open source consultant for various companies
(02:05:45 PM) mdeslaur: And used to build security updates for the defunct Fedora Legacy project
(02:06:35 PM) kees: we're glad to have him as part of the team.  :)
(02:06:43 PM) mdeslaur: uhmm...that's about it I guess, feel free to ask me any question you may have
(02:06:52 PM) jdstrand: *very* glad :)
(02:07:09 PM) kees: so, when there are security updates that need to happen, "ubuntu-security" and "motu-swat" are the ones handling it usually, though we have many additional contributors.
(02:07:30 PM) kees: in general, our "update procedure" is here: https://wiki.ubuntu.com/SecurityUpdateProcedures
(02:07:38 PM) kees: we focus on fixing "CVE"s in published ubuntu releases
(02:07:49 PM) kees: (CVE stands for Common Vulnerabilities and Exposures)
(02:08:02 PM) kees: a CVE is basically a number identifying a flaw in software that has security implications
(02:08:15 PM) kees: the central collection of CVEs here: http://cve.mitre.org/
(02:08:24 PM) kees: some of the URLs for more information that I'm listing here can also be found in our KnowledgeBase: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase
(02:08:34 PM) kees: since CVEs are global identifiers, they cover software (and hardware) from any vendor in the world -- only some CVEs apply to Ubuntu software.
(02:08:46 PM) kees: the first step to fixing security problems in Ubuntu is keeping up to date with new CVEs, and checking to see in Ubuntu is affected.
(02:09:00 PM) kees: 19:08 < bullgard4> QUESTION: What does 'swat' stand for in  "motu-swat"?
(02:09:36 PM) kees: 'swat' is in reference to "Special Weapons and Tactics", a specialize police force
(02:09:53 PM) kees: http://en.wikipedia.org/wiki/SWAT
(02:10:15 PM) jdstrand: I like to think of them as swat'ing security bugs
(02:10:21 PM) kees: that works too :)
(02:10:44 PM) kees: to help coordinate between teams and people, we have an ubuntu-specific reponsitory for tracking CVEs: https://code.launchpad.net/~ubuntu-security/ubuntu-cve-tracker/master
(02:10:56 PM) kees: this is a bzr branch, with details about every CVE that has been issued.
(02:11:05 PM) kees: Once you have a local branch of ubuntu-cve-tracker, the first thing to do is read, surprisingly, the README file.  :)
(02:11:18 PM) kees: most of the CVEs are flagged "ignore" since they apply to unpackaged software, different vendors like Apple or Microsoft, etc.
(02:11:36 PM) kees: since not everyone is interested in digging into a bzr repo just to see how things look, it is also published: http://people.ubuntu.com/~ubuntu-security/cve/main.html
(02:11:41 PM) kees: and universe: http://people.ubuntu.com/~ubuntu-security/cve/universe.html
(02:12:03 PM) kees: and for individual CVEs, those can be examined too, e.g. http://people.ubuntu.com/~ubuntu-security/cve/CVE-2008-2327
(02:12:04 PM) ubot5: kees: Multiple buffer underflows in the (1) LZWDecode, (2) LZWDecodeCompat, and (3) LZWDecodeVector functions in tif_lzw.c in the LZW decoder in LibTIFF 3.8.2 and earlier allow context-dependent attackers to execute arbitrary code via a crafted TIFF file, related to improper handling of the CODE_CLEAR code. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-2327)
(02:12:10 PM) kees: (thanks ubot5)
(02:12:30 PM) kees: once CVEs have been identified, the team will prioritize them, and start either tracking down patches or making our own.
(02:12:50 PM) kees: some CVEs are initially private, while upstream (and the vendors) try to figure out solutions.  This is called an "embargoed" issue.
(02:13:09 PM) kees: the ubuntu tracker only shows public CVEs, since none of the vendors are allowed to discuss embargoed issues until they reach their "coordinated release date"
(02:13:25 PM) kees: once the teams gets a patch to fix a problem, they test them, and finally publish the fixes.
(02:13:34 PM) kees: (I'll be coming back to this in a moment...)
(02:13:43 PM) kees: presently, security updates for main/restricted packages (mostly handled by "ubuntu-security") get "Ubuntu Security Notices" (USNs) published: http://www.ubuntu.com/usn/
(02:13:52 PM) kees: there is a mailing list for this as well: http://lists.ubuntu.com/archives/ubuntu-security-announce/
(02:14:10 PM) kees: the Universe Security Team ("motu-swat") handles updates for universe and multiverse.
(02:14:27 PM) kees: besides the two teams handling security updates, we also have a team dedicated to providing better security globally to Ubuntu.  this is the "ubuntu-hardened" team.
(02:14:40 PM) kees: It started very SELinux-centric, but has grown to include people interested in AppArmor and other hardening techniques.
(02:14:52 PM) kees: we also have a "white hat" team ("ubuntu-whitehat") that is dedicated to hunting down new security issues.  it is still young and has plenty of room for growth.
(02:15:03 PM) kees: since all of the teams are rather small, we still use a single IRC channel (#ubuntu-hardened) and a single mailing list (ubuntu-hardened)
(02:15:15 PM) kees: switching back to update process, once we've tracked down an issue and a fix, we have to test it.  for details on this process, I'll turn it over to jdstrand!
(02:15:38 PM) jdstrand: that's my cue
(02:15:44 PM) jdstrand: Hi! I'm Jamie Strandboge. I am a member of the Ubuntu Security Team and a Canonical employee.
(02:15:56 PM) jdstrand: As many of us know, it is estimated that Ubuntu has 10 million installations world-wide, with only several thousand running the latest development release (10,000 IIRC).
(02:16:10 PM) jdstrand: Users of released versions of Ubuntu expect and deserve a stable system with as few regressions as possible. As such, ensuring high quality updates to released versions of Ubuntu is of utmost importance.
(02:16:31 PM) jdstrand: Hopefully this part of the session will help people to understand the processes, information and tools available to help create high-quality updates.
(02:16:43 PM) jdstrand: Before an update to an Ubuntu release can be made, typically a patch must be created and submitted to Launchpad for review.
(02:16:57 PM) jdstrand: https://wiki.ubuntu.com/StableReleaseUpdates discusses when and how a bug fix can be incorporated into a released version of Ubuntu, while https://wiki.ubuntu.com/SecurityUpdateProcedures discusses the process of fixing security bugs.
(02:17:22 PM) jdstrand: In general, whether the update is a stable release update (SRU) or security update, the updated package should follow the process described in https://wiki.ubuntu.com/SecurityUpdateProcedures#Preparing%20an%20update.
(02:17:35 PM) jdstrand: Most importantly:
(02:17:40 PM) jdstrand: 1. Make the minimum changes necessary to fix the bug
(02:17:47 PM) jdstrand: 2. Prefer packaging the changes as proper patches (ie, when there is a patch system (eg, cdbs, dpatch, quilt, et al), use it. When the patch system supports it, follow https://wiki.ubuntu.com/UbuntuDevelopment/PatchTaggingGuidelines)
(02:18:01 PM) jdstrand: 3. Properly test that the package builds and still works (see qa-regression-testing, discussed later)
(02:18:21 PM) jdstrand: 4. Make sure the changelog has the proper formatting. This includes the proper version, distribution, bug references, patch origin, and CVE. All of this information is needed for review and so it is clear to anyone who looks at the changelog what changed.
(02:18:56 PM) jdstrand: (the CVE is needed only if it's a security update, of course)
(02:19:00 PM) jdstrand: 5. If Debian is still affected, send the patch to them by following https://wiki.ubuntu.com/Debian/Bugs. We need to always give back to Debian, because it is the right thing to do and because it helps Ubuntu in the next development cycle.
(02:19:27 PM) jdstrand: When submitting patches, please read through and follow this section of SecurityUpdateProcedures in its entirety, as the above only touched on the most important points.
(02:19:49 PM) jdstrand: When performing a security update or SRU, it is of utmost importance to make sure that the update does not introduce any regressions and verify that the package works as intended after an update.
(02:20:08 PM) jdstrand: While we are all human and some regressions are inevitable, we can go a long way towards ensuring that we do everything possible to be reasonably sure the update works as intended.
(02:20:19 PM) jdstrand: This is where the QA Regression Testing bzr branch (https://code.launchpad.net/~ubuntu-bugcontrol/qa-regression-testing/master) can help.
(02:20:34 PM) jdstrand: qa-regression-testing was started by Martin Pitt (pitti), and continued by me, kees, mdeslaur and others.
(02:20:48 PM) jdstrand: qa-regression-testing is used extensively by the Ubuntu Security team, as well as the Ubuntu QA Team, Ubuntu Server Team and others. It is also used in the SRU (Stable Release Update) process and when testing Apparmor profiles.
(02:21:02 PM) jdstrand: The bzr branch contains a lot of information to help with an update.
(02:21:13 PM) jdstrand: I highly recommend reading README.testing, which talks about things to look out for in an update, best practices, checklists and more.
(02:21:27 PM) jdstrand: For example, README.testing discusses checking the differences between builds, exercising patched code, running build tests, checking for ABI changes, and more.
(02:21:44 PM) jdstrand: It also lists various software that can be helpful in debugging, analysis, auditing, and ensuring package QA (among other things).
(02:21:57 PM) jdstrand: Finally, it contains a checklist that can be invaluable in making sure that you done/checked various tasks.
(02:22:26 PM) jdstrand: build_testing/ and notes_testing/ have notes and instructions on how to enable build testing, use testing frameworks for a particular application and any other notes pertinent to testing. For example, build_testing/apache2 has notes on using the Perl-Framework from http://httpd.apache.org/test/.
(02:22:48 PM) jdstrand: The scripts/ directory contains scripts for testing various programs. As of today, there are 58 test scripts.
(02:23:02 PM) jdstrand: The main idea behind these scripts is not build/compile testing, but rather application testing for default and non-default configurations of packages.
(02:23:17 PM) jdstrand: For example, the test-openldap.py script will test slapd for various configurations like ldap://, ldapi://, ldaps://, sasl, overlays, different backends and even kerberos integration.
(02:23:38 PM) jdstrand: test-openoffice.org.py opens various files that openoffice.org supports to make sure they still work.
(02:24:26 PM) jdstrand: *IMPORTANT* the scripts in the scripts/ directory are often destructive, and should NOT typically be run on a production machine. We typically run these in a virtual machine, but often a chroot is sufficient.
(02:24:45 PM) jdstrand: Most of the scripts use python-unit. At the top of each script are instructions for how to use it, caveats, etc. There is also a skeleton.py script along with libraries (testlib*.py) that can be used when developing new scripts.
(02:25:05 PM) jdstrand: While python-unit doesn't perfectly lend itself to GUI testing, it may be able to be used to automate some aspects of the testing of certain files, such as test-openoffice.org.py and test-xine.py.
(02:25:31 PM) jdstrand: Work is being done to help automate GUI functionality, and can be seen in https://wiki.ubuntu.com/Testing/Automation and later this week in the 'Automated Desktop Testing' session.
(02:25:50 PM) jdstrand: The scripts in qa-regression-testing typically are written when there is a new security update, and specifically tests the functionality that pertained to a given patch.
(02:26:07 PM) jdstrand: As such, the scripts are in varying states of completeness, and any help in creating and extending these is most welcome. :)
(02:27:03 PM) jdstrand: Help is also welcome in any of the q-r-t parts-- eg build_testing/ information for applications not listed
(02:27:17 PM) jdstrand: By following the checklists, best practices, developing new scripts and using existing scripts for qa-regression-testing, we all can go a long way in helping to ensure as few regressions as possible.
(02:27:42 PM) ***kees <3 q-r-t
(02:27:42 PM) ***jdstrand hands it back over to kees
(02:28:31 PM) kees: the work in q-r-t is very important, so if anyone wants to help with updates, that's by far one of the best places to focus on.
(02:29:03 PM) jdstrand: it also is not hard to get started, and often a fun/small coding project
(02:29:19 PM) kees: I did a quick overview of the various other areas of the security team.  are there any specific questions or areas that people would like to hear more about?
(02:30:54 PM) jdstrand: Something I didn't mention while discussing testing
(02:31:29 PM) jdstrand: as we all know, Ubuntu is split up into different releases, such as dapper, gutsy, hardy, intrepid, and jaunty
(02:32:08 PM) jdstrand: in the changelog, after the version, there is a 'distribution' field
(02:32:32 PM) jdstrand: for the development release (eg 'jaunty'), the distribution field is simply the release name
...