Apport

Differences between revisions 7 and 8
Revision 7 as of 2006-10-23 10:09:59
Size: 7858
Editor: p54B3EDB9
Comment: add internal workflow
Revision 8 as of 2006-10-23 10:24:12
Size: 7890
Editor: c58-107-63-201
Comment: General fixes of spelling/grammar.
Deletions are marked like this. Additions are marked like this.
Line 5: Line 5:
Debugging program crashes without any automated tools has pretty time consuming and hard for both developers and users. Many program crashes remain unreported or unfixed because: Debugging program crashes without any automated tools has been pretty time consuming and hard for both developers and users. Many program crashes remain unreported or unfixed because:
Line 17: Line 17:
 * and presents an UI that informs about the crash and instructs the user how to proceed.  * and presents a UI that informs the user about the crash and instructs them on how to proceed.
Line 21: Line 21:
== How does it look like for users? == == What does it look like for users? ==
Line 25: Line 25:
If any process in the system dies due to a signal that is commonly refered to as 'crash' (segmentation violation, bus error, floating point exception, etc.), the apport backend is automatically invoked. It produces an initial crash report in a file in `/var/crash` (the file name is composed from the name of the crashed executable and the user id). If the crashed process belongs to the user who is currently logged in, apport informs the user about the crash and offers to report the problem: If any process in the system dies due to a signal that is commonly refered to as a 'crash' (segmentation violation, bus error, floating point exception, etc.), the apport backend is automatically invoked. It produces an initial crash report in a file in `/var/crash` (the file name is composed from the name of the crashed executable and the user id). If the crashed process belongs to the user who is currently logged in, apport informs the user about the crash and offers to report the problem:
Line 35: Line 35:
There currently is no automatic way to submit a crash report to the distribution developers (see "Future plans" below). For now, if the user chooses "Report bug", apport-gtk opens the packages' bug tracking page in the web browser and asks the user to create a bug and attach the apport report: There is currently no automatic way to submit a crash report to the distribution developers (see "Future plans" below). For now, if the user chooses "Report bug", apport-gtk opens the packages' bug tracking page in the web browser and asks the user to create a bug and attach the apport report:
Line 39: Line 39:
Experienced can also take a look into the report content: Experienced users can also take a look into the report content:
Line 84: Line 84:
The Ubuntu 6.10 Linux kernel has a patch to call an usermode helper in the event of a crash (in kernel terms: whenever the process would dump core if `ulimit -c` is nonzero). This usermode helper is configured in `/proc/sys/kernel/crashdump-helper`. To avoid cluttering the disk with core dumps, the default ulimit is kept at 0, and instead the kernel introduces a second size limit for temporary core dumps in `/proc/sys/kernel/crashdump-size`. The Ubuntu 6.10 Linux kernel has a patch to call a usermode helper in the event of a crash (in kernel terms: whenever the process would dump its core if `ulimit -c` is nonzero). This usermode helper is configured in `/proc/sys/kernel/crashdump-helper`. To avoid cluttering the disk with core dumps, the default ulimit is kept at 0, and instead the kernel introduces a second size limit for temporary core dumps in `/proc/sys/kernel/crashdump-size`.
Line 90: Line 90:
In order to keep the delay and CPU/IO impact as low as possible, `/usr/share/apport/apport` only collects data which has to be acquired while the crashes process still exists: information from `/proc/`''pid'', the core dump, the executable path, and the signal number. The report is written to `/var/crash/`''executable_path''`.`''uid''`.crash`. In order to keep the delay and CPU/IO impact as low as possible, `/usr/share/apport/apport` only collects data which has to be acquired while the crashed process still exists: information from `/proc/`''pid'', the core dump, the executable path, and the signal number. The report is written to `/var/crash/`''executable_path''`.`''uid''`.crash`.
Line 98: Line 98:
 * Linux kernel 2.6.19 now supports [http://lkml.org/lkml/2006/8/14/78 pipes in core_pattern]. Some issues are currently discussed with upstream, but eventually it would be nice to have apport run on a stock upstream kernel.  * Linux kernel 2.6.19 now supports [http://lkml.org/lkml/2006/8/14/78 pipes in core_pattern]. Some issues are currently being discussed with upstream, but eventually it would be nice to have apport run on a stock upstream kernel.
Line 100: Line 100:
 * Various improvements to performance and utility. See the relevant [https://features.launchpad.net/distros/ubuntu/+spec/apport-improvements specification].  * Various improvements to performance and utilities. See the relevant [https://features.launchpad.net/distros/ubuntu/+spec/apport-improvements specification].

Apport - Automatic crash reports

What is this all about?

Debugging program crashes without any automated tools has been pretty time consuming and hard for both developers and users. Many program crashes remain unreported or unfixed because:

  • Many crashes are not easily reproducible.
  • End users do not know how to prepare a report that is really useful for developers, like building a package with debug symbols, operating gdb, etc.

  • A considerable part of bug triage is spent with collecting relevant information about the crash itself, package versions, hardware architecture, operating system version, etc.
  • There is no easy frontend which allow users to submit detailed problem reports.
  • Existing solutions like bug-buddy or krash are specific to a particular desktop environment, are nontrivial to adapt to the needs of a distribution developer, and do not work for crashes of background servers (like a database or an email server).

Apport is a system which

  • intercepts crashes right when they happen the first time,
  • gathers potentially useful information about the crash and the OS environment,
  • and presents a UI that informs the user about the crash and instructs them on how to proceed.

We hope that this will lead to a much better level of quality assurance in the future.

What does it look like for users?

The user side of apport is designed to be extremely simple and as unannoying as possible.

If any process in the system dies due to a signal that is commonly refered to as a 'crash' (segmentation violation, bus error, floating point exception, etc.), the apport backend is automatically invoked. It produces an initial crash report in a file in /var/crash (the file name is composed from the name of the crashed executable and the user id). If the crashed process belongs to the user who is currently logged in, apport informs the user about the crash and offers to report the problem:

  • attachment:apport-gtk-desktopfile.png

If an user process crashes while the user is not currently logged in, update-notifier will present a notification when the user starts a desktop session the next time:

  • attachment:apport-async-crash.png

Clicking on the bomb icon will cause the same frontend to appear.

There is currently no automatic way to submit a crash report to the distribution developers (see "Future plans" below). For now, if the user chooses "Report bug", apport-gtk opens the packages' bug tracking page in the web browser and asks the user to create a bug and attach the apport report:

  • attachment:apport-gtk-report.png

Experienced users can also take a look into the report content:

  • attachment:apport-gtk-report-details.png

I'm a developer. How to I use these crash reports?

Report format

apport uses the standard Debian control syntax for reports, i. e. keeps everything in a flat file that looks like this:

DistroRelease: Ubuntu 6.10
ExecutablePath: /usr/bin/gcalctool
Package: gcalctool 5.8.24-0ubuntu2
ProcCmdline: gcalctool
ProcEnviron:
 SHELL=/bin/bash
 PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/games
 LANG=de_DE.UTF-8
StackTrace:
 [...]
 #0  0x00002ae577bb37bf in poll () from /lib/libc.so.6
 No symbol table info available.
 #1  0x00002ae57786991e in g_main_context_check () from /usr/lib64/libglib-2.0.so.0
 No symbol table info available.
 [...]
CoreDump: base64
 eJzsXQmcFMXV7+XGA0dBREVoDxSPXQYEB...

Only a tiny subset of the available fields are shown here. Apport reports include a core dump in a compressed and uuencoded format, which is useful for post-mortem debugging and post-mortem generation of a symbolic stack trace.

Tools

There are several tools available for working with a crash report:

  • apport-unpack: Unpack a report into single files (one per attribute). This is most useful for extracting the core dump. Please see the manpage for further details.

  • apport-retrace: Regenerate stack traces of a report. If you supply the -d option, this tool will automatically download available debug symbol packages and use them to generate a symbolic stack trace. The manpage explains the functionality and all available options in detail.

  • python-problem-report: This package ships a Python module problem_report which provides general dictionary access to a crash report and loading/saving methods (not specific to apport reports).

  • python-apport-utils: This package ships a Python module apport_utils which encapsulates core functionality of apport and is specific to crash reports. You can use it to implement your own frontends as well as creating custom crash reports (Ubuntu's graphical installer "ubiquity" uses this).

How does it work internally?

Crash interception

The Ubuntu 6.10 Linux kernel has a patch to call a usermode helper in the event of a crash (in kernel terms: whenever the process would dump its core if ulimit -c is nonzero). This usermode helper is configured in /proc/sys/kernel/crashdump-helper. To avoid cluttering the disk with core dumps, the default ulimit is kept at 0, and instead the kernel introduces a second size limit for temporary core dumps in /proc/sys/kernel/crashdump-size.

apport's init script sets up both files at start, so that /usr/share/apport/apport is called on a crash.

Backend

In order to keep the delay and CPU/IO impact as low as possible, /usr/share/apport/apport only collects data which has to be acquired while the crashed process still exists: information from /proc/pid, the core dump, the executable path, and the signal number. The report is written to /var/crash/executable_path.uid.crash.

Frontend invocation

In Gnome, update-notifier keeps an inotify watch on /var/crash. Whenever there is something new, it calls /usr/share/apport/apport-checkreports. If there are new reports, it calls /usr/share/apport/apport-gtk, which is the frontend shown in the screenshots above.

Future plans

Use the source, Luke!

Apport (last edited 2017-05-25 20:03:48 by penalvch)