Dev Week -- Working with bugs reported by apport -- Brian Murray -- Tue, Jul 12th, 2011

   1 [18:02] <bdmurray> Hi, I'm Brian Murray and I work for Canonical as a defect analyst.  Part of being a defect analyst is working with bug reports in Launchpad.
   2 [18:02] <bdmurray> One of the ways, and actually the preferred way, that bugs are reported to Launchpad is via apport.
   3 [18:03] <bdmurray> Apport is an automated system for reporting problems regarding Ubuntu packages.
   4 [18:03] <bdmurray> These problems include crashes and package installation failures.
   5 [18:04] <bdmurray> As these reports are regularly formatted it is also possible to work with them using automated tools.  This is what I am going to talk about today.
   6 [18:04] <bdmurray> Let's look at bug reported by apport together -
   7 [18:05] <bdmurray> We can tell this bug was one reported by apport because of the keys and values in the description for example ProblemType: Crash and the tag apport-crash.
   8 [18:05] <bdmurray> Both of these indicate that we are looking at a crash reported by apport.  The actual crash appears in Traceback.txt.
   9 [18:06] <bdmurray> The Traceback.txt indicates it is an error importing a python module and since update-manager is installed on a lot of systems this bug is likely to receive a lot of duplicates
  10 [18:06] <bdmurray> We can see the bug already has 10 of them.
  11 [18:06] <bdmurray> Because we already know the cause of the bug and how to fix it but a fix isn't available yet - it'd be a good idea to prevent this bug from being reported any more.
  12 [18:07] <bdmurray> Apport provides us with a system for doing just that - bug patterns.
  13 [18:08] <bdmurray> Are there any questions so far?
  14 [18:09] <bdmurray> So the branch containing bug patterns can be found at
  15 [18:09] <bdmurray> When apport prepares to report a bug about a package it first checks to see if the bug it will report matches a bug pattern.
  16 [18:10] <bdmurray> question from andyrock: how does LP detect bug duplicates?
  17 [18:11] <bdmurray> Duplicate detection is actually done by the apport retracer and it builds a crash signature based of the traceback or stacktrace and then it marks bugs as a duplicate of another.  For an example look at one of the duplicates of bug 807715.
  18 [18:12] <bdmurray> If the bug matches a pattern the reporter will be directed to an existing bug report or a web page instead of going through the bug reporting process.
  19 [18:12] <bdmurray> The pattern   is an xml file that contains pattern matches, using regular expressions, for apport bug keys.
  20 [18:12] <bdmurray> To write one for bug 807715 I used ( the following:
  21 [18:12] <bdmurray> <pattern url="">
  22 [18:13] <bdmurray> This is the url future bug reporters will be directed to.
  23 [18:13] <bdmurray> You could even send them to a wiki page like
  24 [18:13] <bdmurray> ~At this page I've documented the issue people have encountered and provided steps for resolving the issue rather than directing them to a bug report as that may be harder to parse.
  25 [18:14] <bdmurray> Carrying on with the example for bug 807715.  We want the pattern to match a specific package:
  26 [18:14] <bdmurray> <re key="Package">^update-manager </re>
  27 [18:15] <bdmurray> Then we have the unique error that we've encountered.
  28 [18:15] <bdmurray> <re key="Traceback">ImportError: cannot import name GConf</re>
  29 [18:15] <bdmurray> Patterns can even be used to search attachments to the bug report like Traceback.txt as I've done or log files like DpkgTerminalLog.txt.
  30 [18:15] <bdmurray> Anybody from the Ubuntu Bug Control team can commit a bug pattern and I'll happily review any merge proposals.
  31 [18:17] <bdmurray> The bugpatterns.xml file ( contains lots of examples to help you get started.
  32 [18:18] <bdmurray> Are there any questions regarding the format of bugpatterns or how they work?
  33 [18:18] <bdmurray> < andyrock> Question: in <re etc. etc.> `re` means `regular expression`?
  34 [18:18] <bdmurray> yes, that is correct
  35 [18:19] <bdmurray> for example here we can see we are matching either of two specific version of apport
  36 [18:19] <bdmurray> <re key="Package">1.20.1-0ubuntu[4|5]</re>
  37 [18:19] <bdmurray> Any more questions?
  38 [18:20] <bdmurray> Included in the bugpatterns bzr branch are some tools for testing and working with bugpatterns
  39 [18:21] <bdmurray> Once you've written a bug pattern you can test it with the test-local script e.g. ./test-local 807715.
  40 [18:21] <bdmurray> It will download the details from the bug report and reconstruct an apport crash file.
  41 [18:21] <bdmurray> Then the pattern matching function will be run against that crash file.  I usually do this with one or two bug reports to make sure they match my pattern before I commit it.
  42 [18:22] <bdmurray> The bug patterns are auto synced to which is where apport looks for them.
  43 [18:23] <bdmurray> In addition to the test-local script there is a utility called search-bugs which takes a package name and tags as arguments e.g. (search-bugs --package update-manager --tags apport-crash).
  44 [18:23] <bdmurray> This is a great    way to make sure your pattern isn't catching the wrong bug reports.
  45 [18:23] <bdmurray> This is really important as a poorly written bug pattern could end blocking all apport bug reports about package or even all of Ubuntu!
  46 [18:24] <bdmurray> Additionally, while the apport retracer will automatically mark crash reports as duplicates it does not currently do this with package installation failures.
  47 [18:24] <bdmurray> So if you write a bug pattern regarding a package installation failure you can use search-bugs with the '-C' switch to consolidate all the existing bug reports into the one you've identified as the master bug report.
  48 [18:25] <bdmurray> By the way package installation failures are identifiable by "ProblemType: Package" and the tag apport-package.
  49 [18:25] <bdmurray> search-bugs will add a comment to each   bug matching the pattern and mark it as a duplicate of the bug identified in pattern url.
  50 [18:26] <bdmurray> An overview of writing bug patterns can be found at  Are there any questions regarding how to write bug patterns or use the tools in the bug patterns branch?
  51 [18:28] <bdmurray> Another thing worth noting is that the apport retracer also helps identify bug reports that would benefit from a bug pattern by tagging them 'bugpattern-needed'.
  52 [18:28] <bdmurray> It does this for crash reports with more than 10 duplicates.
  53 [18:30] <bdmurray> So we've talked about how we can deal with bugs reported by apport once they've arrived in Launchpad, but what can we do to add information to bugs reported via apport?
  54 [18:31] <bdmurray> pport supports package hooks - meaning that it looks in the directory /usr/share/apport/package-hooks/ for a file matching the name of the binary or the source package and adds the information in the package hook to the bug report.
  55 [18:32] <bdmurray> For example if we look at /usr/share/apport/package-hooks/ which is also viewable at
  56 [18:32] <bdmurray> We can see that we add the file "/var/log/apt/history.log" to the bug report and name the attachment "DpkgHistoryLog.txt".
  57 [18:33] <bdmurray> So now if update-manager crashes or a user reports a bug via 'ubuntu-bug update-manager' we will have an attachment named DpkgHistoryLog.txt added to the bug report.
  58 [18:34] <bdmurray> This is helpful in debugging the issue with update-manager as it provides us with the context for the operation being performed.
  59 [18:35] <bdmurray> The Stable Release Updates team is also happy to approve SRUs that add apport package hooks to a package.
  60 [18:36] <bdmurray> Back to the update-manager package hook - it is utilizing some convenience functions provided by apport.
  61 [18:36] <bdmurray> For example attach_gconf adds the non-default gconf values for update-manager.
  62 [18:37] <bdmurray> We can review the convenience functions at
  63 [18:39] <bdmurray> Useful ones include attach_hardware, command_output, recent_syslog and attach_network.
  64 [18:39] <bdmurray> It is also possible to communicate with the bug reporter in a source package hook by asking questions or displaying messages.
  65 [18:40] <bdmurray> It is even possible to prevent the reporting of bugs - before a bug pattern is ever searched for.
  66 [18:40] <bdmurray> I've done both of these things, asking the reporter a question and preventing certain bugs from being reported, in the ubiquity package hook.
  67 [18:41] <bdmurray> You can see it at
  68 [18:42] <bdmurray>
  69 [18:42] <bdmurray> Here we examine the syslog file from when the installation was run for SQUASHFS errors
  70 [18:43] <bdmurray> In the event that any are found an information dialog is presented to the reporter explaining the issue and informing them about things they can do to resolve the issue.
  71 [18:45] <bdmurray>
  72 [18:45] <bdmurray> Here a Yes / No dialog box is presented to the bug reporter asking them if they want to include their debug log file in the report.
  73 [18:47] <bdmurray> Most ubuntu systems will have lots of package hooks installed on them in /usr/share/apport/package-hooks and there are some great examples in there of things you can do.
  74 [18:47] <bdmurray> The xorg package hook is pretty complicated but gathers lots of information.
  75 [18:49] <bdmurray> The grub2 package hook actually reviews a configuration file for errors which is quite handy.
  76 [18:50] <bdmurray> Are there any questions regarding the material covered so far?
  77 [18:52] <bdmurray> Okay well that's all I have
  78 [18:53] <bdmurray> I hope you understand how you can better work with bugs reported by apport about your package by using bug patterns and how to make those bugs reported by apport even more informative.
  79 [18:53] <bdmurray> Thanks for your time!

MeetingLogs/devweek1107/WorkingWithApportBugs (last edited 2011-07-13 09:25:02 by dholbach)