What is this all about?
Debugging program crashes without any automated tools has been pretty time consuming and hard for both developers and users. Many program crashes remain unreported or unfixed because:
- Many crashes are not easily reproducible.
End users do not know how to prepare a report that is really useful for developers, like building a package with debug symbols, operating gdb, etc.
- A considerable part of bug triage is spent with collecting relevant information about the crash itself, package versions, hardware architecture, operating system version, etc.
- There is no easy frontend which allow users to submit detailed problem reports.
- Existing solutions like bug-buddy or krash are specific to a particular desktop environment, are nontrivial to adapt to the needs of a distribution developer, do not work for crashes of background servers (like a database or an email server), and do not integrate well with existing debug packages that a distribution might provide.
Apport is a system which
- intercepts crashes right when they happen the first time,
- gathers potentially useful information about the crash and the OS environment,
- can be automatically invoked for unhandled exceptions in other programming languages (e. g. in Ubuntu this is done for Python),
- can be automatically invoked for other problems that can be automatically detected (e. g. Ubuntu automatically detects and reports package installation/upgrade failures from update-manager),
- presents a UI that informs the user about the crash and instructs them on how to proceed,
- and is able to file non-crash bug reports about software, so that developers still get information about package versions, OS version etc.
We hope that this will lead to a much better level of quality assurance in the future.
If you want to make crash reports of your software even more useful when being reported through apport, please see /DeveloperHowTo.
What does it look like for users?
The user side of apport is designed to be extremely simple and as unannoying as possible.
If any process in the system dies due to a signal that is commonly referred to as a 'crash' (segmentation violation, bus error, floating point exception, etc.), or e. g. a packaged Python application raises an uncaught exception, the apport backend is automatically invoked. It produces an initial crash report in a file in /var/crash/ (the file name is composed from the name of the crashed executable and the user id). If the crashed process belongs to the user who is currently logged in, or it belongs to a system process and the user is an administrator, apport informs the user about the crash and offers to report the problem:
If a user process crashes while the user is not currently logged in, update-notifier will present a notification when the user starts a desktop session the next time:
Clicking on the icon will cause the same frontend to appear. The notification is also shown for crashes of system processes; since they need the frontend being invoked as root, immediately starting it through gksu would be too disruptive.
Now apport collects various debug information and asks the user what to do with it:
Experienced users can also take a look into the report content:
If the user chooses "Send report", apport-gtk uploads the collected information to the bug tracking system. After that it opens the packages' bug filing page with a sensible default bug title and leaves the rest of bug filing process to the web UI.
How to enable apport
Apport is not enabled by default in stable releases, even if it is installed. There are two ways to enable it.
- If you want to debug a specific program once, just do:
sudo service apport start force_start=1
Then you can simply trigger the crash again, and Apport's dialog will show up with instructions to report a bug with traces. Apport will be automatically disabled on next start.
If you are triaging bugs, this is the best way to get traces from bug reporters that didn't use Apport in the first place.
- To enable it permanently, do this:
sudo nano /etc/default/apport
... and change enabled from "0" to "1".
The automatic crash interception component of apport is disabled by default in stable releases for a number of reasons:
- Apport collects potentially sensitive data, such as core dumps, stack traces, and log files. They can contain passwords, credit card numbers, serial numbers, and other private material.
This is mitigated by the fact that it presents you what will be sent to the bug tracker, and that all crash report bugs are private by default, limited to the Ubuntu bug triaging team. We can reasonably expect developers and technically savvy users, who run the development release, to be aware of this and judge whether it is appropriate to file a crash report. But we shouldn't assume that every Ubuntu user of stable releases is able to do so.
- During the development release we already collect thousands of crash reports, much more than we can ever fix. Continuing to collect those for stable releases is not really useful, since
- The most important crashes have already been discovered in the development release.
The less important ones are not suitable for getting fixed in stable releases (see https://wiki.ubuntu.com/StableReleaseUpdates
- Asking users to send crash reports to us is insincere, since we can't possibly answer and deal with all of them.
- Data collection from apport takes a nontrivial amount of CPU and I/O resources, which slow down the computer and don't allow you to restart the crashed program for several seconds.
Note apport does not trap SIGABRT signals. If you are getting such a signal, then please see DebuggingProgramCrash.
I'm a developer. How do I use these crash reports?
apport internally uses the standard Debian control syntax for reports, i. e. keeps everything in a flat file that looks like this:
DistroRelease: Ubuntu 6.10 ExecutablePath: /usr/bin/gcalctool Package: gcalctool 5.8.24-0ubuntu2 ProcCmdline: gcalctool ProcEnviron: SHELL=/bin/bash PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/games LANG=de_DE.UTF-8 StackTrace: [...] #0 0x00002ae577bb37bf in poll () from /lib/libc.so.6 No symbol table info available. #1 0x00002ae57786991e in g_main_context_check () from /usr/lib64/libglib-2.0.so.0 No symbol table info available. [...] CoreDump: base64 eJzsXQmcFMXV7+XGA0dBREVoDxSPXQYEB...
Only a tiny subset of the available fields are shown here. Apport reports include a core dump in a compressed and unencoded format, which is useful for post-mortem debugging and post-mortem generation of a symbolic stack trace.
However, when uploading the data to a bug tracking system, a different format can be used. e. g. when using Launchpad, the data is uploaded in Multipart/MIME format so that the small parts land directly in the bug summary and the big parts become separate bug attachments.
Some fields warrant further details:
SegvAnalysis: when examining a Segmentation Fault (signal 11), Apport attempts to review the exact machine instruction that caused the fault, and checks the program counter, source, and destination addresses, looking for any virtual memory address (VMA) that is outside an allocated range (as reported in the ProcMaps attachment).
SegvReason: a VMA can be read from, written to, or executed. On a SegFault, one of these 3 CPU actions has taken place at a given VMA that either not allocated, or lacks permissions to perform the action. For example:
SegvReason: reading NULL VMA would mean that a NULL pointer was most likely dereferenced while reading a value.
SegvReason: writing unknown VMA would mean that something was attempting to write to the destination of a pointer aimed outside of allocated memory. (This is sometimes a security issue.)
SegvReason: executing writable VMA [stack] would mean that something was causing code on the stack to be executed, but the stack (correctly) lacked execute permissions. (This is almost always a security issue.)
There are several tools available for working with a crash report:
apport-unpack: Unpack a report into single files (one per attribute). This is most useful for extracting the core dump. Please see the manpage for further details. This tool is not necessary when working with Launchpad, since it already splits the parts into separate attachments.
apport-retrace: Regenerate stack traces of a report. If you supply the -g option, this tool will automatically download available debug symbol packages and use them to generate a symbolic stack trace. The manpage explains the functionality and all available options in detail.
python-problem-report: This package ships a Python module problem_report which provides general dictionary access to a crash report and loading/saving methods (not specific to apport reports).
python-apport: This ships a Python package apport which encapsulates core functionality of apport and is specific to crash and bug reports. You can use it to implement your own frontends and backends.
apport-collect: This checks the source package(s) of an existing Launchpad bug, runs apport hooks for them, and uploads their collected information back to the bug report.
How does it work internally?
Apport uses /proc/sys/kernel/core_pattern directly pipe the core dump into apport. For intercepting Python crashes it installs a /etc/python*/sitecustomize.py to call apport on unhandled exceptions.
In order to keep the delay and CPU/IO impact as low as possible, /usr/share/apport/apport only collects data which has to be acquired while the crashed process still exists: information from /proc/pid, the core dump, the executable path, and the signal number. The report is written to /var/crash/executable_path.uid.crash.
In Gnome, update-notifier keeps an inotify watch on /var/crash. Whenever there is something new, it calls /usr/share/apport/apport-checkreports. If there are new reports, it calls /usr/share/apport/apport-gtk, which is the frontend shown in the screenshots above.
The frontend then collects additional information like package versions, package file checksums, or OS version, and calls all matching package hooks.
The Canonical data center runs a service which automatically retrace bugs with apport. By tagging the bugs according to architecture in Launchpad, a retrace will be done and the tag will be removed. Tags that are used are need-i386-retrace, need-amd64-retrace or need-ppc-retrace. See the announcement.
Per-package Apport Hooks
It is possible for packages to specify information gathered from the system and included in the bug report. These are done by apport hooks contained in packages. For some useful examples see:
- source_xorg.py - adds additional log files and hardware details to bug reports
- usplash - ignores crashes in specific code paths
- source_totem.py - asks the reporter questions and gathers different information based on responses
in /usr/share/apport/package-hooks. There is also a list of packages providing apport hooks.
Please see /DeveloperHowTo for further information.
If a crash or bug report is submitted through apport, the relevant hooks will be run automatically. If you have an already reported bug that was filed without apport, and you are interested in the information from those hooks, you can ask the bug reporter to use apport-collect bugnumber (see #Tools).
Use the source, Luke!
apport is developed with the bazaar RCS on Launchpad. If you want to contribute to it or develop your own system based on it, you can get your own branch with bzr get lp:apport for trunk, or debcheckout -a apport for the Ubuntu packaging branch.
You can also browse it online.
Various improvements to performance, better tools to work with reports, and integration of more languages (Mono/Python stack traces, assertion messages, etc.) See the relevant specification.
Please do not hesitate to report bugs and feature requests to the bug tracker.