CrashdumpRecipe

Differences between revisions 1 and 23 (spanning 22 versions)
Revision 1 as of 2010-07-21 14:14:28
Size: 4236
Editor: jeremyfoshee
Comment:
Revision 23 as of 2016-02-03 21:59:01
Size: 7868
Editor: arges
Comment:
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:
"The LKCD (Linux Kernel Crash Dump) project is a set of kernel patches and utilities to allow a copy of the kernel memory to be saved in the event of a kernel panic. The saved kernel image makes forensics on the kernel panic possible with utilities included in the package. Most commercial Unix operating systems come with similar crash utilities, but this package is fairly new to Linux and has to be added on manually. The LKCD utility is not designed to gather helpful information in the case of a hardware caused panic or a segment violation. The complete LKCD package is available for download at http://lkcd.sourceforge.net/."
Line 4: Line 3:
For convenience, the kernel crash dump utility has been packaged in Ubuntu. It can be installed with the following command: ||<tablestyle="float:right; font-size: 0.9em; width:40%; background:#F1F1ED; margin: 0 0 1em 1em;" style="padding:0.5em;"><<TableOfContents>>||

= Introduction =

The Ubuntu Kernel Crash Dump is a mechanism that enable enterprise style post-mortem crash analysis in Linux operating systems. It uses a special mode of kexec which allows to automatically boot a secondary kernel whenever a crash (Oops/panic) occurs. This secondary kernel will then save the state and memory of the primary kernel to a certain location of the filesystem (''/var/crash'' on newer releases). This file can then be used by '''crash''' to gather detailed information about the problem.

= Installation =

For convenience, the kernel crash dump utility has been packaged in Ubuntu. It can be installed with the following command: {{{
sudo apt-get install linux-crashdump }}}

Newer versions of the package will automatically add an entry ''crashkernel=384M-2G:64M,2G-:128M'' to the kernel commandline in grub. However this may cause problems on systems with less than 2G of memory (see [[#Troubleshooting|troubleshooting]]).

= Verifying linux-crashdump installation =

For Trusty, please see [[https://help.ubuntu.com/lts/serverguide/kernel-crash-dump.html|here]].

= Inspecting the crash dump using crash =

In order to use the generated crash dump with '''crash''' one needs the ''vmlinux'' file which has the debugging information. This is part of the kernel ddeb package which can be found at:

[[http://ddebs.ubuntu.com/pool/main/l/linux/]]
Line 7: Line 27:
 apt-get install linux-crashdump sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-proposed main restricted universe multiverse
EOF

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C01
sudo apt-get update
sudo apt-get install linux-image-$(uname -r)-dbgsym
Line 10: Line 39:
== Ubuntu 9.10 "Karmic Koala" == /!\ Be aware that those packages are huge! (~600 MB)
Line 12: Line 41:
In Karmic all that is needed is to install the "linux-crashdump" package. After a reboot the system should be able to catch crash dumps automatically and provide them to apport. When installed, the debug kernel can be found under ''/usr/lib/debug/boot/'' and '''crash''' is started by: {{{
crash <debug kernel> <crash dump> }}}
Line 14: Line 44:
For example, to test you can force a kernel oops: Unfortunately the tool does not allow to look at a 32bit dump on a 64bit system and the other way round. Also it tends to be quite picky about matching up kernel and dump.

= Inspecting the crash dump using apport-retrace =

To get a local retrace, you need apport-retrace and then run: {{{
apport-retrace --stdout --rebuild-package-info /var/crash/linux-image*.crash }}}

/!\ Again, this can take a while because it needs to download the kernel debug package.

== Enabling various types of panics ==

To make Linux kernel to panic on different situations please use:
Line 16: Line 58:
 echo 1 > /proc/sys/kernel/panic_on_oops
 echo c > /proc/sysrq-trigger
}}}
This should force a kernel oops and automatic reboot. Then watch for an apport prompt in the notification area on the next login.

To get a local retrace, you need apport-retrace and then run:
{{{
# apport-retrace --stdout --rebuild-package-info /var/crash/linux-image*.crash
}}}
(this can take a while because it needs to download the linux-image-debug package and that file is several hundreds megs).

To do the backtrace manually, you you have to install "crash" (ie linux-crashdump) and the linux-image-debug-`uname -r` kernel debug deb package from ddebs.ubuntu.com. Note, you can run the apport-retrace command above which will also unpack and install the linux-image-debug-`uname -r` kernel debug deb package. Then you need to get the VmCore from apport again and use "crash" with all its power. Try the following commands:
{{{
# apport-unpack /var/crash/linux-image*.crash /tmp/unpacked
# crash /usr/lib/debug/boot/vmlinux-`uname -r` /tmp/unpacked/vmcore
crash> bt -a
echo 1 > /proc/sys/kernel/hung_task_panic # panic when hung task is detected
echo 1 > /proc/sys/kernel/panic_on_io_nmi # panic on NMIs from I/O
echo 1 > /proc/sys/kernel/panic_on_oops # panic on oops or kernel bug detection
echo 1 > /proc/sys/kernel/panic_on_unrecovered_nmi # panic on NMIs from memory or unknown
echo 1 > /proc/sys/kernel/softlockup_panic # panic when soft lockups are detected
echo 1 > /proc/sys/vm/panic_on_oom # panic when out-of-memory happens
Line 34: Line 66:
Note: the linux-image-debug-* packages do not exist in the usual repositories - you have to use download the packages from http://ddebs.ubuntu.com/pool/main/l/linux/. See https://bugs.launchpad.net/ubuntu/+source/linux/+bug/289087, https://lists.ubuntu.com/archives/kernel-team/2009-February/004310.html, https://lists.ubuntu.com/archives/kernel-team/2009-March/004570.html, https://lists.ubuntu.com/archives/kernel-team/2009-June/005931.html = Troubleshooting =
Line 36: Line 68:
== Ubuntu 9.04 "Jaunty Jackalope" == == Allocated memory for the crash kernel ==
Line 38: Line 70:
This page describes a recipe for enabling crash dump vmcore analysis on your Jaunty x86/x86_64 platform. Much of the information was gleaned from the kernel source tree files in Documentation/kdump. When testing crash dump sometimes the system just seems to lock up. The main issue there is how much memory was assigned for the crash kernel. When kexec starts the crash kernel it requires enough memory to fit the unpacked kernel, the compressed initrd and the uncompressed initrd (at least while unpacking). If there is not enough memory allocated, things usually go wrong without any hint. To solve this there are the following options:
Line 40: Line 72:
  * 'apt-get install linux-crashdump'
    This is a meta package that installs all of the tools necessary to acquire and analyse a crash-dump vmcore.
 1. Increase the allocation by changing ''crashkernel='' on the grub command line or in ''/boot/grub/grub.cfg'' (for grub2) or ''/boot/grub/menu.lst'' (for old grub). To avoid loosing the settings when running '''update-grub''' the change can be made in ''/etc/grub.d/10_linux''.
 1. Reduce the size of the initrd. By default this is set to include all the modules and firmware ever needed. This allows using the same initrd on any system but increases its size a lot. In order to limit it to the modules really required to boot on the current hardware, change the following in ''/etc/initramfs-tools/initramfs.conf'': {{{
 ...
 MODULES=dep
 ... }}}
Line 43: Line 78:
  * Add 'crashkernel=64M@16M' to the kernel command line in /boot/grub/menu.lst.
    You'll also probably want to remove 'quiet splash'.
== Crash kernel fails to load: Hang ==
Line 46: Line 80:
  * Reboot the system (into the ordinary kernel). The section of RAM above will now be reserved for the crashkernel (and not available to the normal system). This can be frustrating to debug, especially if you're unable to record the console messages from the new kexec kernel. A serial console attached to the system is best here to continue debugging. An easy troubleshooting step is to systematically eliminate the additional kernel parameters passed to the crash kernel and retrying. These arguments are kept in '''/etc/init.d/kdump''': {{{
...
        # Append kdump_needed for initramfs to know what to do, and add
        # maxcpus=1 to keep things sane.
        APPEND="$APPEND kdump_needed maxcpus=1 irqpoll reset_devices"
Line 48: Line 86:
  * Make note of your root partition, e.g., /dev/sda1
    'kexec -p /boot/vmlinuz-{{{`uname -r`}}} --initrd=/boot/initrd.img-{{{`uname -r`}}} --append="root=<ROOT_PARTITION> irqpoll maxcpus=1"'
    This loads the crash-dump kernel into the reserved memory, in preparation for a panic.
        # --elf32-core-headers is needed for 32-bit systems (ok
        # for 64-bit ones too).
        log_action_begin_msg "Loading crashkernel"
        kexec -p "$KERNEL_IMAGE" --initrd="$INITRD" --append="$APPEND"
        log_action_end_msg $?
... }}}
Line 52: Line 93:
  Now your kernel is ready to acquire a post-crash vmcore. You can test the process by simulating a crash-dump: Leave '''$APPEND''' and '''kdump_needed'''. Start by removing '''reset_devices''' and then
install the new kexec crash kernel configuration: {{{
sudo service kdump start }}}
Line 54: Line 97:
  'echo c > /proc/sysrq-trigger' Then retest; if that doesn't work, remove the next argument, rinse and repeat.
Line 56: Line 99:
  What you should see is a boot sequence, which is the crash dump kernel loading. Login as root and copy /proc/vmcore to a location of your choice, e.g. cp /proc/vmcore /var/log/vmcore.
  Reboot back to the normal kernel and use crash to analyse the vmcore:
== ACPI memory hotplug issues ==
Line 59: Line 101:
  'crash /boot/System.map-{{{`uname -r`}}} /lib/modules/{{{`uname -r`}}}/vmlinux /var/log/vmcore If you see the following call trace from your serial console after kexecing into the crash kernel you may need to append 'acpi_no_memhotplug' to the crashdump kernel cmdline.
Line 61: Line 103:
  The methods used for examining the vmcore using crash are left as an exercise for the user. {{{
Call Trace:
 dump_stack+0x45/0x57
 warn_alloc_failed+0xf2/0x140
 __alloc_pages_nodemask+0x2e4/0xa10
 vmemmap_alloc_block+0xb5/0xbf
 vmemmap_alloc_block_buf+0x15/0x3b
 vmemmap_populate+0xb3/0x20c
 sparse_mem_map_populate+0x29/0x38
 sparse_add_one_section+0x71/0x16e
 __add_pages+0xb9/0x280
 arch_add_memory+0x71/0xf0
 add_memory+0xdf/0x210
 acpi_memory_device_add+0x1ab/0x282
 acpi_bus_attach+0xe3/0x196
 acpi_bus_scan+0x70/0x8f
 acpi_scan_init+0x89/0x1d3
 acpi_init+0x272/0x28f
 do_one_initcall+0xb3/0x200
 kernel_init_freeable+0x17b/0x21a
 kernel_init+0xe/0xe0
 ret_from_fork+0x3f/0x70
}}}

Edit KDUMP_CMDLINE_APPEND in /etc/default/kdump-tools such that it is un-commented and contains 'acpi_no_memhotplug' as well. Then restart the kdump service.

= Release specific notes =

== Ubuntu 12.04 "Precise Pangolin" ==

 * [[https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/785394|Bug 785394: Hard-coded crashkernel=... memory reservation in /etc/grub.d/10_linux is insufficient]]<<BR>>
 The default allocation for systems below 2G is not enough for the current initrd size. Manually adapting the size allows to use the crash kernel.
 * The current (1.3.7-2) version of makedumpfile reports to be incompatible with the 3.2 kernel. The dumps created seem to be ok.

== Ubuntu 15.10 "Wily Werewolf" and later ==
 * [[https://bugs.launchpad.net/ubuntu/+source/kexec-tools/+bug/1496317|Bug 1496317: kexec fails with OOM killer with the current crashkernel=128 value]]<<BR>>
The current allocation for the crashkernel value is too low to correctly load the default initrd.img. This means that the OOM killer will break the crash dump capture procedure. While the bug is being worked on, you can increase the value of crashkernel to something more than 150Mb to work around the bug.

Introduction

The Ubuntu Kernel Crash Dump is a mechanism that enable enterprise style post-mortem crash analysis in Linux operating systems. It uses a special mode of kexec which allows to automatically boot a secondary kernel whenever a crash (Oops/panic) occurs. This secondary kernel will then save the state and memory of the primary kernel to a certain location of the filesystem (/var/crash on newer releases). This file can then be used by crash to gather detailed information about the problem.

Installation

For convenience, the kernel crash dump utility has been packaged in Ubuntu. It can be installed with the following command:

sudo apt-get install linux-crashdump 

Newer versions of the package will automatically add an entry crashkernel=384M-2G:64M,2G-:128M to the kernel commandline in grub. However this may cause problems on systems with less than 2G of memory (see troubleshooting).

Verifying linux-crashdump installation

For Trusty, please see here.

Inspecting the crash dump using crash

In order to use the generated crash dump with crash one needs the vmlinux file which has the debugging information. This is part of the kernel ddeb package which can be found at:

http://ddebs.ubuntu.com/pool/main/l/linux/

sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)          main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-proposed main restricted universe multiverse
EOF

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C01
sudo apt-get update
sudo apt-get install linux-image-$(uname -r)-dbgsym

Warning /!\ Be aware that those packages are huge! (~600 MB)

When installed, the debug kernel can be found under /usr/lib/debug/boot/ and crash is started by:

crash <debug kernel> <crash dump> 

Unfortunately the tool does not allow to look at a 32bit dump on a 64bit system and the other way round. Also it tends to be quite picky about matching up kernel and dump.

Inspecting the crash dump using apport-retrace

To get a local retrace, you need apport-retrace and then run:

apport-retrace --stdout --rebuild-package-info /var/crash/linux-image*.crash 

Warning /!\ Again, this can take a while because it needs to download the kernel debug package.

Enabling various types of panics

To make Linux kernel to panic on different situations please use:

echo 1 > /proc/sys/kernel/hung_task_panic          # panic when hung task is detected
echo 1 > /proc/sys/kernel/panic_on_io_nmi          # panic on NMIs from I/O
echo 1 > /proc/sys/kernel/panic_on_oops            # panic on oops or kernel bug detection
echo 1 > /proc/sys/kernel/panic_on_unrecovered_nmi # panic on NMIs from memory or unknown
echo 1 > /proc/sys/kernel/softlockup_panic         # panic when soft lockups are detected
echo 1 > /proc/sys/vm/panic_on_oom                 # panic when out-of-memory happens

Troubleshooting

Allocated memory for the crash kernel

When testing crash dump sometimes the system just seems to lock up. The main issue there is how much memory was assigned for the crash kernel. When kexec starts the crash kernel it requires enough memory to fit the unpacked kernel, the compressed initrd and the uncompressed initrd (at least while unpacking). If there is not enough memory allocated, things usually go wrong without any hint. To solve this there are the following options:

  1. Increase the allocation by changing crashkernel= on the grub command line or in /boot/grub/grub.cfg (for grub2) or /boot/grub/menu.lst (for old grub). To avoid loosing the settings when running update-grub the change can be made in /etc/grub.d/10_linux.

  2. Reduce the size of the initrd. By default this is set to include all the modules and firmware ever needed. This allows using the same initrd on any system but increases its size a lot. In order to limit it to the modules really required to boot on the current hardware, change the following in /etc/initramfs-tools/initramfs.conf:

     ...
     MODULES=dep
     ... 

Crash kernel fails to load: Hang

This can be frustrating to debug, especially if you're unable to record the console messages from the new kexec kernel. A serial console attached to the system is best here to continue debugging. An easy troubleshooting step is to systematically eliminate the additional kernel parameters passed to the crash kernel and retrying. These arguments are kept in /etc/init.d/kdump:

...
        # Append kdump_needed for initramfs to know what to do, and add
        # maxcpus=1 to keep things sane.
        APPEND="$APPEND kdump_needed maxcpus=1 irqpoll reset_devices"

        # --elf32-core-headers is needed for 32-bit systems (ok
        # for 64-bit ones too).
        log_action_begin_msg "Loading crashkernel"
        kexec -p "$KERNEL_IMAGE" --initrd="$INITRD" --append="$APPEND"
        log_action_end_msg $?
... 

Leave $APPEND and kdump_needed. Start by removing reset_devices and then install the new kexec crash kernel configuration:

sudo service kdump start 

Then retest; if that doesn't work, remove the next argument, rinse and repeat.

ACPI memory hotplug issues

If you see the following call trace from your serial console after kexecing into the crash kernel you may need to append 'acpi_no_memhotplug' to the crashdump kernel cmdline.

Call Trace:
 dump_stack+0x45/0x57
 warn_alloc_failed+0xf2/0x140
 __alloc_pages_nodemask+0x2e4/0xa10
 vmemmap_alloc_block+0xb5/0xbf
 vmemmap_alloc_block_buf+0x15/0x3b
 vmemmap_populate+0xb3/0x20c
 sparse_mem_map_populate+0x29/0x38
 sparse_add_one_section+0x71/0x16e
 __add_pages+0xb9/0x280
 arch_add_memory+0x71/0xf0
 add_memory+0xdf/0x210
 acpi_memory_device_add+0x1ab/0x282
 acpi_bus_attach+0xe3/0x196
 acpi_bus_scan+0x70/0x8f
 acpi_scan_init+0x89/0x1d3
 acpi_init+0x272/0x28f
 do_one_initcall+0xb3/0x200
 kernel_init_freeable+0x17b/0x21a
 kernel_init+0xe/0xe0
 ret_from_fork+0x3f/0x70

Edit KDUMP_CMDLINE_APPEND in /etc/default/kdump-tools such that it is un-commented and contains 'acpi_no_memhotplug' as well. Then restart the kdump service.

Release specific notes

Ubuntu 12.04 "Precise Pangolin"

Ubuntu 15.10 "Wily Werewolf" and later

The current allocation for the crashkernel value is too low to correctly load the default initrd.img. This means that the OOM killer will break the crash dump capture procedure. While the bug is being worked on, you can increase the value of crashkernel to something more than 150Mb to work around the bug.

Kernel/CrashdumpRecipe (last edited 2016-07-17 10:44:24 by shalzz95)