Ubuntu Open Week - Verifying Stable Update (SRU) bugfixes - Steve Beattie - Thu, Nov 6th, 2008
(01:03:58 PM) sbeattie: Alright, let's get started. Questions in #ubuntu-classroom-chat please, feel free to ask as we go along. (01:04:55 PM) sbeattie: I'm from the ubuntu QA team, here to talk a little bit about the process we go through to verify bugfixes for updates. (01:05:22 PM) sbeattie: First, a little overview of the stable release updates (SRU) process (01:06:07 PM) sbeattie: SRUs are updates to packages in released versions of the distro (e.g. Intrepid, Hardy) to fix particularly annoying bugs (01:06:57 PM) sbeattie: The process is that we get a proposed fix for a bug from a developer or upstream. (01:07:47 PM) sbeattie: Once we have that, either the SRU team (for packages in main) or MOTU SRU (for universe) will approve the fix, then the archive admins will upload the package to the aprropriate -proposed repository (01:08:48 PM) sbeattie: Once that happens, then verification of the fix occurs, and the package can go into the updates repository (01:09:14 PM) sbeattie: The annoying cdrom eject behavior in intrepid is being fixed via this process, for example. (01:09:41 PM) sbeattie: The process is described in detail at https://wiki.ubuntu.com/StableReleaseUpdates (01:10:33 PM) sbeattie: Security updates go through a slightly different but similar process, due to often being embargoed such that they can't be publicly discussed until a certain date (01:12:06 PM) sbeattie: There are two things we're looking for when a package moves into -proposed before it can go into -updates: verification of the fix itself and regressions (01:12:40 PM) sbeattie: Both are important to find; we want to find regressions so that we don't break things for existing users (01:13:37 PM) sbeattie: But we also want to verify that the fix does actually improve the situation for our users; if it doesn't help we're wasting their time and bandwidth as well as ours. (01:14:18 PM) sbeattie: The current set of outstanding SRUs are tracked (unfortunately) in 3 different locations: (01:14:31 PM) sbeattie: The master page at http://people.ubuntu.com/~ubuntu-archive/pending-sru.html (01:14:50 PM) sbeattie: A page specifically dedicated to packages in main at http://people.ubuntu.com/~sbeattie/sru_todo.html (01:15:08 PM) sbeattie: And a corresponding one for universe at http://qa.ubuntuwire.com/sru/todo.html (01:15:47 PM) sbeattie: (each contains slightly different information, which is why they all exist, but my intention is to merge them all eventually) (01:16:18 PM) sbeattie: The process we follow when verifying is roughly as follows: (01:16:43 PM) sbeattie: - set up a known, relatively pristine, environment (01:17:09 PM) sbeattie: - try to reproduce the issue in the most recent published version of the software (01:17:38 PM) sbeattie: - enable the -proposed repository and install the proposed update for that software (01:17:56 PM) sbeattie: - try to reproduce the issue with the updated software (01:18:11 PM) sbeattie: - report results on the SRU bug report (01:18:37 PM) sbeattie: I'll cover these steps in a little more detail with a walkthrough of a particular bug (01:19:16 PM) sbeattie: The first step is to make sure we have a relatively pristine version of the distro the fix is targeted for (01:19:53 PM) sbeattie: we want to ensure we eliminate outside effects, and are just seeing the changes due to the package update (01:20:22 PM) sbeattie: virtualization is useful here, particularly with snapshotting capabilities (01:20:54 PM) sbeattie: so kvm, virtualbox, xen, vmware et al are nice tools to have for an isolated environment (01:21:42 PM) sbeattie: Generally, we also want to make sure the image is up-to-date with regard to updates (01:22:06 PM) sbeattie: Then we try to reproduce the bug in question. (01:22:28 PM) sbeattie: We want to do this so that we know what we're testing for when we install the proposed package. (01:23:06 PM) sbeattie: Often just doing this will offer us more insights into the actual problem being solved than just a cursory read of the bug report. (01:23:44 PM) sbeattie: If we can't reproduce the initial problem, we have much less confidence that the update actually is going to fix the issue (01:24:20 PM) sbeattie: (sometimes, the bug has already been fixed and launchpad's status is out of date for whatever reason) (01:25:02 PM) sbeattie: hardware specific bugs offer their own special problems; if we don't have the hardware, it's much harder to reproduce (01:25:51 PM) sbeattie: There's not much we can do except prompt on the bug report for people affected by the bug to do the testing and install the package ourselves to look for regressions. (01:26:23 PM) sbeattie: These are often kernel or X problems, though webcams are particular issue for intrepid (01:27:06 PM) sbeattie: The example bug we'll look at is an SRU for update-manager in intrepid, bug 40058 (01:27:15 PM) sbeattie: https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/40058 (01:27:50 PM) sbeattie: update-manager is a key tool, the default gui for performing updates; it's important we don't break this for our users! (01:28:21 PM) sbeattie: The bug is that changelogs aren't being shown to users (01:28:44 PM) sbeattie: particularly if source repositories aren't enabled in apt (01:29:18 PM) sbeattie: In this case, the dev has been nice enough to put in an explicit TESTCASE section (01:30:02 PM) sbeattie: That's not always the case, so sometimes you get to figure out how to test for the condition (01:31:29 PM) sbeattie: Normally, we would wait to enable the -proposed repository until after we've reproduced the bug, but in this case we'll do it now as part of the reproduction procedure (01:32:11 PM) sbeattie: You can do this through the menus, via System -> Admin -> Software Sources menu (01:32:35 PM) sbeattie: or invoke the tool directly 'sudo software-properties-gtk' (01:33:10 PM) sbeattie: From there, under the 'updates' tab menu, tick the "proposed updates" box (01:33:35 PM) sbeattie: alternately, you can edit or script changes to your /etc/apt/sources.list file directly (01:34:06 PM) sbeattie: you'll want to ensure that apt's information is update, via 'sudo apt-get update' (01:35:24 PM) sbeattie: (If you're specifically editing /etc/apt/sources.list, you'd want something similar to "deb http://us.archive.ubuntu.com/ubuntu/ intrepid-proposed universe main multiverse restricted" as a line in there) (01:36:09 PM) sbeattie: Again, we'd normally do this after reproducing the bug on the original software. (01:36:39 PM) sbeattie: As part of reproducing, we want to double-check exactly what version of the software we're testing. (01:37:09 PM) sbeattie: In this case, we'll look to see what version of update-manager and update-manager-core we have installed (01:37:35 PM) sbeattie: from the command line, we can do: apt-cache policy update-manager update-manager-core (01:37:59 PM) sbeattie: The relevant portions of that are: (01:38:02 PM) sbeattie: Version table: (01:38:02 PM) sbeattie: 1:0.93.34 0 (01:38:02 PM) sbeattie: 500 http://us.archive.ubuntu.com intrepid-proposed/main Packages (01:38:02 PM) sbeattie: *** 1:0.93.32 0 (01:38:02 PM) sbeattie: 500 http://us.archive.ubuntu.com intrepid/main Packages (01:38:02 PM) sbeattie: 100 /var/lib/dpkg/status (01:38:23 PM) sbeattie: (note that we're also seeing the proposed version as available but not installed_ (01:38:41 PM) sbeattie: (the *** indicates which version is installed) (01:39:11 PM) sbeattie: Okay, so we've got the original version installed, let's run it and see if we can reproduce (01:39:38 PM) sbeattie: we'll open update-manager, either via the menu or via 'sudo update-manager' directly (01:40:08 PM) sbeattie: scroll (if necessary) to the proposed updates section (01:40:57 PM) sbeattie: if you're not seeing some proposed updates available, then either something's gone wrong in the process of enabling the -proposed repository, or you already had -proposed enabled (01:41:31 PM) sbeattie: Tick the 'description of update' toggle for one of the proposed updates and you should see something like (01:41:38 PM) sbeattie: http://people.ubuntu.com/~sbeattie/OW2008/Update_Manager_unfixed.png (01:42:10 PM) sbeattie: If so, then yay, we've reproduced the bug. (01:42:51 PM) sbeattie: so then we move on to seeing if the bug still exists in the proposed version. (01:43:15 PM) sbeattie: we'll go ahead and install the update-manager and update-manager-core packages from proposed (01:44:00 PM) sbeattie: again, we don't want to install all the packages in proposed here, because we want isolate interference to ensure we know what we're looking at (01:45:09 PM) sbeattie: since we're testing update-manager we could untick all the other updates, but exiting and running 'sudo apt-get install update-manager update-manager-core' is a bit quicker (01:45:37 PM) sbeattie: then we try to reproduce the problem again (01:46:17 PM) sbeattie: as before, we want to run 'apt-cache policy update-manager update-manager-core' to ensure we have the right packages installed (01:47:17 PM) sbeattie: Now version 1:0.93.34 of each should be installed (01:47:29 PM) sbeattie: we'll run update-manager again (01:47:55 PM) sbeattie: and look at the details for some of the proposed updates (01:48:11 PM) sbeattie: should see something like: http://people.ubuntu.com/~sbeattie/OW2008/Update_Manager_proposed.png (01:48:43 PM) sbeattie: Yay, looks like the bug is fixed in the proposed package. (01:49:28 PM) sbeattie: We'll poke around checking out the details for multiple packages to make sure they're all showing changelogs (01:49:52 PM) sbeattie: Once we've done that, we can look for regressions as well. (01:50:12 PM) sbeattie: Some things to look for in terms of regresions: (01:50:28 PM) sbeattie: - install requirements; does it depend on other packages in proposed, particularly ones not from the same source package? (01:50:54 PM) sbeattie: if it does, and both packages aren't moved into updates at the same time, the package will be uninstallable. (01:51:34 PM) sbeattie: - possible differences in behavior for fresh install versus update from buggy version; sometimes the fixes only show up for new installations, particularly for packaging problems (01:51:52 PM) sbeattie: Things to do: (01:52:11 PM) sbeattie: just generally play with the app; helps here if you're familiar with it. (01:52:44 PM) sbeattie: For update-manager we'll go ahead and install some updates, and see if things still work properly (01:53:05 PM) sbeattie: we can also look on https://testcases.qa.ubuntu.com/ for things to test (01:53:18 PM) sbeattie: The security team has some regression scripts in a bzr tree: (01:53:18 PM) sbeattie: https://code.launchpad.net/~ubuntu-bugcontrol/qa-regression-testing/master (01:53:59 PM) sbeattie: Finally, you can also enable the proposed repository for your day-to-day desktop: https://wiki.ubuntu.com/Testing/EnableProposed (01:54:30 PM) sbeattie: That comes at some risk, if a bad update comes through. But we want to know about that as early as possible! (01:54:56 PM) sbeattie: Finally we report our results on the SRU bug on launchpad (01:55:14 PM) sbeattie: Both failures and successes are important to report! (01:55:30 PM) sbeattie: note both the before and after versions of the software you tested (01:55:51 PM) sbeattie: also report anything out of the ordinary or unexpected you had to do in testing (01:56:28 PM) sbeattie: it's also helpful to update the description of the bug if elements are missing (eg. you created a TESTCASE) (01:57:17 PM) sbeattie: When doing all this, if you run into problems, you can always solicit for help on #ubuntu-bugs and #ubuntu-testing, as well as asking directly on the bug report (01:57:40 PM) sbeattie: We could use help in doing verifications and looking for regressions (01:58:06 PM) sbeattie: In particular, hardy SRUs have lagged recently due to focus on the intrepid release (01:58:30 PM) sbeattie: There's an SRU verification team on launchpad at https://launchpad.net/~sru-verification (01:58:59 PM) sbeattie: And this coming monday, November 10th, we'll be focusing a testing day on SRU verifications as well. (01:59:26 PM) sbeattie: That's all I have, if there's quick questions I can take them here or over in #ubuntu-testing (02:02:02 PM) jcastro: ok thanks sbeattie!