Ubuntu Wiki

Dev Week -- Creating high-quality updates -- ubuntu-security -- Tue, Jan 20

UTC -4 (EST)

(02:01:09 PM) kees: I'll go ahead and get started.  As usual, please ask questions at any time in the -chat room, and we'll answer them as we see them.  :)
(02:01:28 PM) kees: welcome everyone!  I'm Kees Cook, and I'm the technical lead of the Ubuntu Security Team (and employed by Canonical).
(02:01:38 PM) kees: this is going to be a quick intro to how the Security Team operates within Ubuntu, and how people can help create high-quality updates.
(02:02:00 PM) kees: I'm joined by Jamie Strandboge and Marc Deslauriers, who will be helping with the presentation.
(02:02:15 PM) kees: the main wiki page for information about the team is here: https://wiki.ubuntu.com/SecurityTeam
(02:02:35 PM) kees: our page is still a bit young, so pardon the lack of details in the FAQ and KnowledgeBase areas.
(02:02:55 PM) kees: we might be able to populate some of the FAQ with today's classroom's questions.  :)
(02:03:14 PM) jdstrand: \o/
(02:03:36 PM) kees: the most active subteams within the Security Team are "ubuntu-security" and "motu-swat"
(02:04:00 PM) kees: 19:03 < ScottK> QUESTION: Since Marc is new(ish) perhaps he could introduce himself ...
(02:04:13 PM) mdeslaur: yes, I guess I could :)
(02:04:43 PM) mdeslaur: I'm from Quebec City, Canada, and have been a Canonical employee since the beginning of November
(02:05:04 PM) mdeslaur: I used to be a security and open source consultant for various companies
(02:05:45 PM) mdeslaur: And used to build security updates for the defunct Fedora Legacy project
(02:06:35 PM) kees: we're glad to have him as part of the team.  :)
(02:06:43 PM) mdeslaur: uhmm...that's about it I guess, feel free to ask me any question you may have
(02:06:52 PM) jdstrand: *very* glad :)
(02:07:09 PM) kees: so, when there are security updates that need to happen, "ubuntu-security" and "motu-swat" are the ones handling it usually, though we have many additional contributors.
(02:07:30 PM) kees: in general, our "update procedure" is here: https://wiki.ubuntu.com/SecurityUpdateProcedures
(02:07:38 PM) kees: we focus on fixing "CVE"s in published ubuntu releases
(02:07:49 PM) kees: (CVE stands for Common Vulnerabilities and Exposures)
(02:08:02 PM) kees: a CVE is basically a number identifying a flaw in software that has security implications
(02:08:15 PM) kees: the central collection of CVEs here: http://cve.mitre.org/
(02:08:24 PM) kees: some of the URLs for more information that I'm listing here can also be found in our KnowledgeBase: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase
(02:08:34 PM) kees: since CVEs are global identifiers, they cover software (and hardware) from any vendor in the world -- only some CVEs apply to Ubuntu software.
(02:08:46 PM) kees: the first step to fixing security problems in Ubuntu is keeping up to date with new CVEs, and checking to see in Ubuntu is affected.
(02:09:00 PM) kees: 19:08 < bullgard4> QUESTION: What does 'swat' stand for in  "motu-swat"?
(02:09:36 PM) kees: 'swat' is in reference to "Special Weapons and Tactics", a specialize police force
(02:09:53 PM) kees: http://en.wikipedia.org/wiki/SWAT
(02:10:15 PM) jdstrand: I like to think of them as swat'ing security bugs
(02:10:21 PM) kees: that works too :)
(02:10:44 PM) kees: to help coordinate between teams and people, we have an ubuntu-specific reponsitory for tracking CVEs: https://code.launchpad.net/~ubuntu-security/ubuntu-cve-tracker/master
(02:10:56 PM) kees: this is a bzr branch, with details about every CVE that has been issued.
(02:11:05 PM) kees: Once you have a local branch of ubuntu-cve-tracker, the first thing to do is read, surprisingly, the README file.  :)
(02:11:18 PM) kees: most of the CVEs are flagged "ignore" since they apply to unpackaged software, different vendors like Apple or Microsoft, etc.
(02:11:36 PM) kees: since not everyone is interested in digging into a bzr repo just to see how things look, it is also published: http://people.ubuntu.com/~ubuntu-security/cve/main.html
(02:11:41 PM) kees: and universe: http://people.ubuntu.com/~ubuntu-security/cve/universe.html
(02:12:03 PM) kees: and for individual CVEs, those can be examined too, e.g. http://people.ubuntu.com/~ubuntu-security/cve/CVE-2008-2327
(02:12:04 PM) ubot5: kees: Multiple buffer underflows in the (1) LZWDecode, (2) LZWDecodeCompat, and (3) LZWDecodeVector functions in tif_lzw.c in the LZW decoder in LibTIFF 3.8.2 and earlier allow context-dependent attackers to execute arbitrary code via a crafted TIFF file, related to improper handling of the CODE_CLEAR code. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-2327)
(02:12:10 PM) kees: (thanks ubot5)
(02:12:30 PM) kees: once CVEs have been identified, the team will prioritize them, and start either tracking down patches or making our own.
(02:12:50 PM) kees: some CVEs are initially private, while upstream (and the vendors) try to figure out solutions.  This is called an "embargoed" issue.
(02:13:09 PM) kees: the ubuntu tracker only shows public CVEs, since none of the vendors are allowed to discuss embargoed issues until they reach their "coordinated release date"
(02:13:25 PM) kees: once the teams gets a patch to fix a problem, they test them, and finally publish the fixes.
(02:13:34 PM) kees: (I'll be coming back to this in a moment...)
(02:13:43 PM) kees: presently, security updates for main/restricted packages (mostly handled by "ubuntu-security") get "Ubuntu Security Notices" (USNs) published: http://www.ubuntu.com/usn/
(02:13:52 PM) kees: there is a mailing list for this as well: http://lists.ubuntu.com/archives/ubuntu-security-announce/
(02:14:10 PM) kees: the Universe Security Team ("motu-swat") handles updates for universe and multiverse.
(02:14:27 PM) kees: besides the two teams handling security updates, we also have a team dedicated to providing better security globally to Ubuntu.  this is the "ubuntu-hardened" team.
(02:14:40 PM) kees: It started very SELinux-centric, but has grown to include people interested in AppArmor and other hardening techniques.
(02:14:52 PM) kees: we also have a "white hat" team ("ubuntu-whitehat") that is dedicated to hunting down new security issues.  it is still young and has plenty of room for growth.
(02:15:03 PM) kees: since all of the teams are rather small, we still use a single IRC channel (#ubuntu-hardened) and a single mailing list (ubuntu-hardened)
(02:15:15 PM) kees: switching back to update process, once we've tracked down an issue and a fix, we have to test it.  for details on this process, I'll turn it over to jdstrand!
(02:15:38 PM) jdstrand: that's my cue
(02:15:44 PM) jdstrand: Hi! I'm Jamie Strandboge. I am a member of the Ubuntu Security Team and a Canonical employee.
(02:15:56 PM) jdstrand: As many of us know, it is estimated that Ubuntu has 10 million installations world-wide, with only several thousand running the latest development release (10,000 IIRC).
(02:16:10 PM) jdstrand: Users of released versions of Ubuntu expect and deserve a stable system with as few regressions as possible. As such, ensuring high quality updates to released versions of Ubuntu is of utmost importance.
(02:16:31 PM) jdstrand: Hopefully this part of the session will help people to understand the processes, information and tools available to help create high-quality updates.
(02:16:43 PM) jdstrand: Before an update to an Ubuntu release can be made, typically a patch must be created and submitted to Launchpad for review.
(02:16:57 PM) jdstrand: https://wiki.ubuntu.com/StableReleaseUpdates discusses when and how a bug fix can be incorporated into a released version of Ubuntu, while https://wiki.ubuntu.com/SecurityUpdateProcedures discusses the process of fixing security bugs.
(02:17:22 PM) jdstrand: In general, whether the update is a stable release update (SRU) or security update, the updated package should follow the process described in https://wiki.ubuntu.com/SecurityUpdateProcedures#Preparing%20an%20update.
(02:17:35 PM) jdstrand: Most importantly:
(02:17:40 PM) jdstrand: 1. Make the minimum changes necessary to fix the bug
(02:17:47 PM) jdstrand: 2. Prefer packaging the changes as proper patches (ie, when there is a patch system (eg, cdbs, dpatch, quilt, et al), use it. When the patch system supports it, follow https://wiki.ubuntu.com/UbuntuDevelopment/PatchTaggingGuidelines)
(02:18:01 PM) jdstrand: 3. Properly test that the package builds and still works (see qa-regression-testing, discussed later)
(02:18:21 PM) jdstrand: 4. Make sure the changelog has the proper formatting. This includes the proper version, distribution, bug references, patch origin, and CVE. All of this information is needed for review and so it is clear to anyone who looks at the changelog what changed.
(02:18:56 PM) jdstrand: (the CVE is needed only if it's a security update, of course)
(02:19:00 PM) jdstrand: 5. If Debian is still affected, send the patch to them by following https://wiki.ubuntu.com/Debian/Bugs. We need to always give back to Debian, because it is the right thing to do and because it helps Ubuntu in the next development cycle.
(02:19:27 PM) jdstrand: When submitting patches, please read through and follow this section of SecurityUpdateProcedures in its entirety, as the above only touched on the most important points.
(02:19:49 PM) jdstrand: When performing a security update or SRU, it is of utmost importance to make sure that the update does not introduce any regressions and verify that the package works as intended after an update.
(02:20:08 PM) jdstrand: While we are all human and some regressions are inevitable, we can go a long way towards ensuring that we do everything possible to be reasonably sure the update works as intended.
(02:20:19 PM) jdstrand: This is where the QA Regression Testing bzr branch (https://code.launchpad.net/~ubuntu-bugcontrol/qa-regression-testing/master) can help.
(02:20:34 PM) jdstrand: qa-regression-testing was started by Martin Pitt (pitti), and continued by me, kees, mdeslaur and others.
(02:20:48 PM) jdstrand: qa-regression-testing is used extensively by the Ubuntu Security team, as well as the Ubuntu QA Team, Ubuntu Server Team and others. It is also used in the SRU (Stable Release Update) process and when testing Apparmor profiles.
(02:21:02 PM) jdstrand: The bzr branch contains a lot of information to help with an update.
(02:21:13 PM) jdstrand: I highly recommend reading README.testing, which talks about things to look out for in an update, best practices, checklists and more.
(02:21:27 PM) jdstrand: For example, README.testing discusses checking the differences between builds, exercising patched code, running build tests, checking for ABI changes, and more.
(02:21:44 PM) jdstrand: It also lists various software that can be helpful in debugging, analysis, auditing, and ensuring package QA (among other things).
(02:21:57 PM) jdstrand: Finally, it contains a checklist that can be invaluable in making sure that you done/checked various tasks.
(02:22:26 PM) jdstrand: build_testing/ and notes_testing/ have notes and instructions on how to enable build testing, use testing frameworks for a particular application and any other notes pertinent to testing. For example, build_testing/apache2 has notes on using the Perl-Framework from http://httpd.apache.org/test/.
(02:22:48 PM) jdstrand: The scripts/ directory contains scripts for testing various programs. As of today, there are 58 test scripts.
(02:23:02 PM) jdstrand: The main idea behind these scripts is not build/compile testing, but rather application testing for default and non-default configurations of packages.
(02:23:17 PM) jdstrand: For example, the test-openldap.py script will test slapd for various configurations like ldap://, ldapi://, ldaps://, sasl, overlays, different backends and even kerberos integration.
(02:23:38 PM) jdstrand: test-openoffice.org.py opens various files that openoffice.org supports to make sure they still work.
(02:24:26 PM) jdstrand: *IMPORTANT* the scripts in the scripts/ directory are often destructive, and should NOT typically be run on a production machine. We typically run these in a virtual machine, but often a chroot is sufficient.
(02:24:45 PM) jdstrand: Most of the scripts use python-unit. At the top of each script are instructions for how to use it, caveats, etc. There is also a skeleton.py script along with libraries (testlib*.py) that can be used when developing new scripts.
(02:25:05 PM) jdstrand: While python-unit doesn't perfectly lend itself to GUI testing, it may be able to be used to automate some aspects of the testing of certain files, such as test-openoffice.org.py and test-xine.py.
(02:25:31 PM) jdstrand: Work is being done to help automate GUI functionality, and can be seen in https://wiki.ubuntu.com/Testing/Automation and later this week in the 'Automated Desktop Testing' session.
(02:25:50 PM) jdstrand: The scripts in qa-regression-testing typically are written when there is a new security update, and specifically tests the functionality that pertained to a given patch.
(02:26:07 PM) jdstrand: As such, the scripts are in varying states of completeness, and any help in creating and extending these is most welcome. :)
(02:27:03 PM) jdstrand: Help is also welcome in any of the q-r-t parts-- eg build_testing/ information for applications not listed
(02:27:17 PM) jdstrand: By following the checklists, best practices, developing new scripts and using existing scripts for qa-regression-testing, we all can go a long way in helping to ensure as few regressions as possible.
(02:27:42 PM) ***kees <3 q-r-t
(02:27:42 PM) ***jdstrand hands it back over to kees
(02:28:31 PM) kees: the work in q-r-t is very important, so if anyone wants to help with updates, that's by far one of the best places to focus on.
(02:29:03 PM) jdstrand: it also is not hard to get started, and often a fun/small coding project
(02:29:19 PM) kees: I did a quick overview of the various other areas of the security team.  are there any specific questions or areas that people would like to hear more about?
(02:30:54 PM) jdstrand: Something I didn't mention while discussing testing
(02:31:29 PM) jdstrand: as we all know, Ubuntu is split up into different releases, such as dapper, gutsy, hardy, intrepid, and jaunty
(02:32:08 PM) jdstrand: in the changelog, after the version, there is a 'distribution' field
(02:32:32 PM) jdstrand: for the development release (eg 'jaunty'), the distribution field is simply the release name
(02:32:54 PM) jdstrand: for security updates, you should append -security to the release name, eg hardy-security
(02:33:15 PM) jdstrand: for SRUs, you should append -proposed (updates are tested in -proposed first)
(02:33:46 PM) jdstrand: many people know about all that, however, what is less widely known is how security updates and SRUs end up getting built
(02:34:36 PM) jdstrand: when testing an update destined for -security (eg, hardy-security), keep in mind that security updates are *only* built with the release pocket and the security pocket enabled
(02:34:58 PM) jdstrand: specifically, -updates and -proposed are not in sources.list when a -security update is being built
(02:35:41 PM) jdstrand: this is important to note when testing, because sometimes subtle (and not so subtle) bugs can creep in when you test your package with -updates enabled, but upload to -security
(02:36:00 PM) jdstrand: which doesn't have -updates enabled (they are essentially different packages)
(02:36:36 PM) jdstrand: another thing to keep in mind, is what components a package is built with
(02:36:59 PM) jdstrand: the components are main, restricted, universe and multiverse
(02:37:20 PM) jdstrand: a package destined for main is only built with the main component
(02:37:25 PM) jdstrand: restricted is built with only main and restricted
(02:37:29 PM) jdstrand: universe is built with main and universe
(02:37:35 PM) jdstrand: multiverse is built with main, restricted, universe and multiverse
(02:38:08 PM) jdstrand: why is this important? sometimes debian/control has a Build-Depends like (foo | bar)
(02:38:58 PM) jdstrand: if foo is in universe and bar is in main, a package destined for main will be built with bar, but if the universe component was in sources.list during your testing, you might have tested with the package built against foo
(02:39:40 PM) jdstrand: QUESTION: How do backports and security updates fit in with each other?  Do we patch the original released or the backported version?
(02:39:55 PM) jdstrand: good question
(02:40:32 PM) jdstrand: -backports is a different pocket entirely, and does not receive official support from the Security Team, and often doesn't receive backported fixes from motu-swat
(02:41:30 PM) jdstrand: updated packages that fix a CVE will pull from -updates though
(02:42:00 PM) jdstrand: so the source will come from -updates, but will be built against only -security (and release)
(02:42:36 PM) jdstrand: it is general policy to backport fixes to the packages that exist at release time, rather than doing an upload of a new version
(02:43:24 PM) jdstrand: for example, if package foo is at version 1.2.3, and upstream releases version 1.2.4 (that has a CVE fix), we will backport the security fix in 1.2.4 to 1.2.3, rather than upgrading foo to 1.2.4
(02:43:42 PM) jdstrand: there are exceptions though-- firefox, postgresql and clamav are some of them
(02:44:03 PM) jdstrand: QUESTION: What is "the release pocket and the security pocket"?
(02:44:24 PM) jdstrand: the release pocket is simply the release name, eg "jaunty"
(02:45:11 PM) jdstrand: after a version of Ubuntu is official, then the release pocket is frozen, and updates must go to other pockets (these can be seen in your /etc/apt/sources.list file)
(02:45:54 PM) jdstrand: so, "hardy" is the release pocket for hardy, "hardy-security" gets security only fixes, and "hardy-updates" gets bug fixes (and security fixes are copied into it)
(02:46:23 PM) jdstrand: "hardy-proposed" is the pocket people upload to when they want a bug fix ultimately destined for "hardy-updates"
(02:47:45 PM) jdstrand: QUESTION: You are doing your best to create high-quality updates. Where is laid down what an Ubuntu user should do to have a very secure Ubuntu computer and network?
(02:48:58 PM) jdstrand: Ubuntu by default is installed with no open ports
(02:49:27 PM) jdstrand: as long as you keep your system up to date, the typical Ubuntu user will have a secure system
(02:50:04 PM) jdstrand: once you start adding services to it, like a web server, Ubuntu can keep the package up to date, but misconfiguration can always be a problem
(02:50:54 PM) jdstrand: there are various tools such as apparmor and ufw that can help lock down services and network capabilities, but this is really a question that can't be sufficiently answered in this session
(02:51:15 PM) jdstrand: QUESTION: If -security is built without -updates, what happens  when you need to prepare a security update to a package which has  an update already published in -updates?
(02:51:53 PM) jdstrand: we pull the version in upddates, patch the security fix in it, and change the version such that it is higher than in -updates
(02:55:35 PM) jdstrand: we are about out of time. are there any more questions?
(02:57:40 PM) jdstrand: Ok. thanks to kees and to everyone who attended. Please feel free to jump in with patches for security, SRUs or testing scripts :)
(02:57:47 PM) jdstrand: happy hacking!
(02:58:09 PM) ickis: nice talk
(02:58:14 PM) ickis: thanks!
(02:58:22 PM) jdstrand: thanks you :)