The main challenge the Mozilla Team faces is processing automatic crash reports. To be useful, the raw core dumps are converted into fully symbolized stack traces. The stack traces are analyzed to identify issue signatures. Other stack traces are reviewed for similar signatures to identify duplicates.

Creating Retraces

Creating symbolized stack retraces is currently done manually via apport-retrace. To create a good retrace, there are a number of conditions that must be met:

1. The system running retrace must match the reporter's system. All executables and shared libraries touched by the core dump must match.
2. Debug symbols, in the form of -dbgsym files, must be available for all of the packages in 1.

Once these conditions are met, apport-retrace generates the symbolized retrace via gdb.

Meeting these conditions present a number of problems. Retrace must:

1. Have a complete coredump as part of the crash report.
2. Know what versions of packages the reporter has installed, including shared libraries.
3. Find .debs for of the correct version of all packages in 2. This is particularly problematic when the report requires a older version of a package then is in the repository.
4. Find .dbgsyms for of the .debs in 3. Same problems with historical packages.

Martin Pitt is working on the above problems.

Automatic Retraces

Retraces should be created as a service in the data center.

1. Manually running retraces is time consuming, labor intensive, and bandwidth intensive.
2. There are potential security issues when storing core dumps on a public server.

A solution is to catch crash reports as they are uploaded to Launch Pad. Store the reports on a secure server. Run apport-retrace as a service. Then place the sanitized crash information on LP for triaging.

1. Ideally crashes would be handled via a separate crash database. Please see wiki.ubuntu.com/CrashReporting for more informaion.

No one is currently working on these problems

Analyzing Retraces

Symbolized retraces must then be manually analyzed by an experienced developer to identify the source of the problem.

No trivial solution to above problem.

Searching for Duplicate Crashes

After the experienced developer identifies a bug signature, other retraces are searched for similarities. Searching is either done manually by triagers or with bughelper. Bughelper is a tool that can scan reports for individual strings.

Having triagers scan reports manually is very time consuming. Bughelp has good potential for automatic report scanning.

The BugSquad is developing bughelper.

Bughelper

Bughelper is in the early stages of development. The current problems are:

1. It is difficult to add new clues.
2. Runs on the client side.
3. Runs in batch mode rather than as an event.

New clues must be added manually. Running bughelper on the client side is time, space, and bandwidth intensive for large data sets. Currently, Bughelper downloads and scans each issue individually on the client. This requires several hours because of the hand shaking. 10 Gigabytes of local cache space is required to store issue attachments. Therefore, it currently works most effective to run it as a daily cron job.

There are number of ways to make bughelper more useful for searching reports.

1. Run as a service in the data center.
2. Run as an event that is triggered when a new crash report is submitted.
3. Create a web interface for adding or modifying clues. Also triggers new search.

No one is currently working on these problems

Downstream

A number of distributions based on Ubuntu have expressed interest in being able to work more closely with LP and Ubuntu on issue handling. These interests include:

1. Extending LP to send updates of an issue's status to a downstream bugzilla via email.
2. Extending apport to send crash information to bugzilla via email.
3. Extending apport to directly interact with bugzilla via web interface.
4. Extending LP to act as crash retrace service for distributions that do not have the resources to run a data center.

Other

1. Link to package specific pages when filing bug.
2. Display all comments for duplicate issues on a single page.

CategoryMozillaTeam