Ubuntu Wiki

Breakpad is a client-side library used by Google Chrome and Mozilla Firefox for error reporting.

While the reports generated by Breakpad are best handled by Google and Mozilla themselves, it is worthwhile for us to collect them too. Without the error data from Firefox and Chrome, our calculation for the average errors per calendar day will be inaccurate, unless we consider it to be the average number of errors that we can directly fix. We will also not know what the stability of these browsers is relative to other applications in Ubuntu.

Breakpad's tools are not currently packaged for Ubuntu. To get the minidump_walk binary, which will translate a minidump-formatted report into a stack trace, branch Breakpad from svn:

svn checkout http://google-breakpad.googlecode.com/svn/trunk/ ~/google-breakpad
cd ~/google-breakpad; ./configure

Just like a regular binary application crash on Ubuntu, Chrome and Firefox are stripped of symbols needed to produce a complete stack trace. For Firefox, these live in the firefox-mozsymbols package, which produces a 21MB zip file:

sudo apt-get install firefox-mozsymbols
mkdir /tmp/syms
cd /tmp/syms
unzip /usr/lib/firefox/firefox-18.0.en-US.linux-x86_64.crashreporter-symbols.zip

With these in place, we can now produce a full stack trace:

DISPLAY=:0 firefox &; PID="$\!"; sleep 3; kill -SEGV $PID
cd ~/google-breakpad
./src/processor/minidump_stackwalk ~/.mozilla/firefox/Crash\ Reports/pending/*.dmp /tmp/syms

We can generate something akin to the Stacktrace Address Signature we produce client-side to avoid having to send up a full report. When minidump_stackwalk is used without a symbols directory, it produces a library plus offset block for each frame:

Thread 0 (crashed)
0  libc-2.15.so + 0xe8303
...

Generating a SAS is not as critical for Breakpad reports as it was for regular binary application crashes. A sample Breakpad report from Firefox is only 638K.

With the complete stacktrace, we can move on to obtaining the rest of the metadata for the crash and turning all this data into an Apport report. The metadata can be obtained by running minidump_dump over the report:

/src/processor/minidump_dump ~/.mozilla/firefox/Crash\ Reports/pending/*.dmp

This contains sections for the process environment, status fields, mapped libraries, etc.

Some additional metadata lives in a .extra file in the same directory as the minidump report. Its format is key-value pairs, separated by = and newline characters. This contains information about which addons were installed, what URL was open at the time of the crash, and some other less-interesting keys.