CrashReporting

Summary

Currently apport sends crash reports as public bugs in Launchpad. We will improve this process to avoid exposing potentially sensitive data to the public and avoid sending unwanted bug email to developers.

Because a few bugs cause most crashes, this system should eventually involve a database of crash reports, automatically aggregated by type so that developers can allocate their time to the top crashers.

Release note

Bug reports about crashes which are automatically generated by the Apport system are now private by default. This avoids exposing potentially sensitive data like passwords to the public. Those bug reports are now inspected by a trusted Ubuntu developer before marking it public.

Rationale

The original idea was to create an entirely separate crash database and web frontend for it. However, since we need a lot of the features of a bug tracker (assignments, release tracking, searching, duplication, email/web interaction, getting feedback from reporters), this database will just use Malone in an appropriate way.

Use cases

  • Willy was creating a logo for his soccer club in Inkscape when it crashed. Being the family's Ubuntu expert, he feels a responsibility to help improve the system. He's reported one or two bugs in Launchpad before, though it wasn't a particularly enjoyable experience.
  • Millie was logging in to her bank account when Firefox crashed. She's used to clicking the "Send" button for Windows Error Reporting, so she does the same for apport. Her banking password can be found in the crash information, so this needs to be treated confidentially.

Design

Authenticated bug filing

We currently do not want bugs to be filed anonymously, because:

  • we want to keep the possibility of contacting the reporters for further information,
  • the current testing community is large enough to supply us with much more reports that the triagers can handle anyway (and enough bugs to keep the reprocessing infrastructure busy), and
  • mostly this would just result in even more duplicates.

New approach of crash bug reporting workflow

  1. Create Launchpad users which control access to the bug, split by main/restricted and universe/multiverse.

  2. Apport files bugs as private/nonsecurity by default, with only the crash reprocessing bot subscribed (Launchpad user apport, the "Apport retracing service").

  3. The retracing bot generates the symbolic stack traces, handles duplicates (see apport-crash-duplicates), and removes the core dump attachment. Then it subscribes the relevant team to the bug.

  4. The triaging teams regularly inspect crash reports (preferably prioritized by number of duplicates). After verifying that a stack trace does not contain sensitive information, they can set the bug to "public".

Comparisons to other crash feedback agents

Implementation

  1. Create two new Launchpad teams ubuntu-crashes-main (for components main and restricted) and ubuntu-crashes-universe (for components universe and multiverse), set a "black hole" contact email address and put the ubuntu-testing teams into them. This avoids sending crash email (explicit requirement from our desktop team) and allows for later adjustments for accessing crash bugs.

  2. Malone currently has the ability to set tags in the blob that gets uploaded to the cloakroom. This needs to get extended to mark a bug as private/nonsecurity (see #116367).

  3. (design is self-explanatory)
  4. (design is self-explanatory)

Test/Demo plan

  1. Cause a packaged program to crash:

    • To generate a signal crash, simply do bash -c 'kill -SEGV $$'.

    • To generate a Python crash, edit the program to have an assert False or similarly disruptive instruction anywhere and run it.

  2. Apport will notify you about the crash and offer to file it. Do that.

  3. The bug will be private and only accessible by you and the LP user apport (Apport retracing service).

  4. After a while the retracing bots will notice the bug, check it for duplicates (for Python crashes) or retrace it/attach symbolic stack traces/check for duplicate (for signal crashes) and subscribe ubuntu-crashes-main for a package from main or restricted, and ubuntu-crashes-universe for other packages. The bug should not have a CoreDump.gz attachment any more, unless the retrace failed and the bug was tagged with apport-failed-retrace.

Comments

MatthewPaulThomas:

  • "We currently do not want bugs to be filed anonymously" because we get "more reports tha[n] the triagers can handle anyway"? That's circular reasoning. If they were crash reports rather than bug reports, they wouldn't need triaging! Similarly, "this would just result in even more duplicates" wouldn't be a problem, it would be a feature. Developers would instead start with the most-commonly-occurring crashes and report bugs about them, like developers do with other crash-reporting systems.
  • Requiring the person experiencing a crash to get a Launchpad account will bias the results. For example, it will mean Edubuntu-specific software (and other kids' software) is under-represented, because the people using it don't even know what a "bug" is, or may even be prevented from signing up.

MartinPitt responds:

  • I am fully aware that this is not the ultimate solution. Eventually Malone should get a concept of a crash database. This spec is meant to be an intermediate solution for the worst problems we currently have, using the tools that are available today (except fr point 2).

Hggdh:

  • I think MatthewPaulThomas has a good point. Requiring LaunchPad membership has the potential to restrict bug reporting... and it seems it would be better for us to have bugs reported even if the reporter is not willing to become a LP member.

    We may want to contact the reporter, so asking for feedback email address (à la Bug Buddy) makes more sense than requiring LaunchPad membership. Nevertheless, the bug should be accepted -- always.

    We might add in a warning stating something like: we urge you to become a LaunchPad member, or at least to provide an email contact address: bug analysis and resolution may require us to request more details from you. If we are unable to do so, we may have no other option but to close the bug without resolution.

MartinPitt responds:

Please read what I said. LP will get a proper crash database, that has already been discussed and is a matter of time. But until we do not have that, we simply cannot provide anonymous bug filing. This spec is the best we can do with the current tools.

BrianJMurrell:

  • What about the (user-)unknown disclosure of information that sending an apport crash exposes? How many users do you think understand that sending an evolution (full) crash report includes sending everything needed to access their e-mail accounts (i.e. in the core file)? That is of course just one of many many examples of information disclosure that most users will not understand they are engaging in.

    • MartinPitt: I'm afraid there is no really great solution to this. We mitigate this by filing bugs privately. The core dump is normally only visible to the apport retracer, which deletes the attachment after it finished successfully. However, we do keep the core dump around if the retracing failed, so that people in ubuntu-testing can access it (which is a reasonably trusted group, but still quite a large number). Bugs are never made public automatically.

    How about an option in apport to exclude sending the core file, probably worded in some more generic "don't send portions of the report that may include sensitive information" kind of prose. It is worth noting that even stack traces can contain a password, and I agree that sending a report without stack traces is pretty useless.
    • A similar option already exists. If the stack trace on the client machine is helpful, we offer the option to send a 'reduced' report which does not contain the core dump. We are aware of the possibility of sensitive information in stack traces, that's why we file bugs privately and ask users to confirm that they really want to send us potentially sensitive information (I know, most users cannot decide this properly, but if they are afraid they can just say 'no').

    Perhaps something along the lines of scrash could help. I don't think anything came of that paper but perhaps other projects exist to try to achieve the same goals of data scrubbing.

CategorySpec