CrashdumpRecipe

Revision 2 as of 2010-07-26 17:44:56

Clear message

"The LKCD (Linux Kernel Crash Dump) project is a set of kernel patches and utilities to allow a copy of the kernel memory to be saved in the event of a kernel panic. The saved kernel image makes forensics on the kernel panic possible with utilities included in the package. Most commercial Unix operating systems come with similar crash utilities, but this package is fairly new to Linux and has to be added on manually. The LKCD utility is not designed to gather helpful information in the case of a hardware caused panic or a segment violation. The complete LKCD package is available for download at http://lkcd.sourceforge.net/."

For convenience, the kernel crash dump utility has been packaged in Ubuntu. It can be installed with the following command: 

 apt-get install linux-crashdump

 

Ubuntu 10.04 "Lucid Lynx"

Same as 9.10, except note that apport-retrace is broken - see:

On the other hand, apport-update+crash seem to work fine, as described for 9.10.

Ubuntu 9.10 "Karmic Koala"

In Karmic all that is needed is to install the "linux-crashdump" package. After a reboot the system should be able to catch crash dumps automatically and provide them to apport.

For example, to test you can force a kernel oops: 

 echo 1 > /proc/sys/kernel/panic_on_oops
 echo c > /proc/sysrq-trigger

This should force a kernel oops and automatic reboot. Then watch for an apport prompt in the notification area on the next login.

To get a local retrace, you need apport-retrace and then run: 

# apport-retrace --stdout --rebuild-package-info /var/crash/linux-image*.crash

(this can take a while because it needs to download the linux-image-debug package and that file is several hundreds megs).

To do the backtrace manually, you you have to install "crash" (ie linux-crashdump) and the linux-image-debug-uname -r kernel debug deb package from ddebs.ubuntu.com. Note, you can run the apport-retrace command above which will also unpack and install the linux-image-debug-uname -r kernel debug deb package. Then you need to get the VmCore from apport again and use "crash" with all its power. Try the following commands: 

# apport-unpack /var/crash/linux-image*.crash /tmp/unpacked
# crash /usr/lib/debug/boot/vmlinux-`uname -r` /tmp/unpacked/vmcore
crash> bt -a

Note: the linux-image-debug-* packages do not exist in the usual repositories - you have to use download the packages from http://ddebs.ubuntu.com/pool/main/l/linux/. See https://bugs.launchpad.net/ubuntu/+source/linux/+bug/289087, https://lists.ubuntu.com/archives/kernel-team/2009-February/004310.html, https://lists.ubuntu.com/archives/kernel-team/2009-March/004570.html, https://lists.ubuntu.com/archives/kernel-team/2009-June/005931.html

Ubuntu 9.04 "Jaunty Jackalope"

This page describes a recipe for enabling crash dump vmcore analysis on your Jaunty x86/x86_64 platform. Much of the information was gleaned from the kernel source tree files in Documentation/kdump.

  • 'apt-get install linux-crashdump'
    • This is a meta package that installs all of the tools necessary to acquire and analyse a crash-dump vmcore.
  • Add 'crashkernel=64M@16M' to the kernel command line in /boot/grub/menu.lst.
    • You'll also probably want to remove 'quiet splash'.
  • Reboot the system (into the ordinary kernel). The section of RAM above will now be reserved for the crashkernel (and not available to the normal system).
  • Make note of your root partition, e.g., /dev/sda1

    • 'kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initrd.img-`uname -r` --append="root=<ROOT_PARTITION> irqpoll maxcpus=1"' This loads the crash-dump kernel into the reserved memory, in preparation for a panic.

    Now your kernel is ready to acquire a post-crash vmcore. You can test the process by simulating a crash-dump:

    'echo c > /proc/sysrq-trigger' What you should see is a boot sequence, which is the crash dump kernel loading. Login as root and copy /proc/vmcore to a location of your choice, e.g. cp /proc/vmcore /var/log/vmcore. Reboot back to the normal kernel and use crash to analyse the vmcore:

    'crash /boot/System.map-`uname -r` /lib/modules/`uname -r`/vmlinux /var/log/vmcore The methods used for examining the vmcore using crash are left as an exercise for the user.