Jump to content
Search

The new Ubuntu Wiki is live. Legacy content may be unavailable and page links may have changed. Read the announcement on Discourse.

Debugging kernel panics remotely with netconsole

From Ubuntu Wiki

Introduction

Netconsole is a kernel module for logging console messages over UDP to a separate machine, useful for capturing failures remotely when capturing logs is difficult on the system under test.

Netconsole has some limitations:

  • A separate receiver machine is required in order to capture the UDP packets sent from the system under test. If you have no other systems running Linux, you can boot Ubuntu from a live USB temporarily to complete the capture.
  • Netconsole initializes when modules are loaded into the kernel. This doesn't allow capture of failures during early boot.
  • The driver for the network card must support the netpoll API.
  • Only IP networks, UDP packets, and Ethernet devices are supported.

Set up netconsole

This document will guide you through the steps to set up netconsole for Ubuntu.

Step 1: Determine remote MAC address

We need to know the MAC address of the receiving machine.

When the receiver is in same subnet

To send the UDP packets to 192.168.1.103, which in this example is the same subnet as the sender, you would find the MAC address with the following: 

pet@sender:~$ ping -c 1 192.168.1.103 > /dev/null
pet@sender:~$ arp -n 192.168.1.103
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.103            ether   08:00:46:d4:1d:82   C                     eth0

Therefore, 08:00:46:d4:1d:82 is the MAC address of the receiver.

When the receiver isn't in the same subnet

First, determine the default gateway IP address: 

pet@sender:~$ netstat -rn | grep ^0.0.0.0
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0

In this case, that's 192.168.1.1.

Next, find the MAC address of the default gateway: 

pet@sender:~$ ping -c 1 192.168.1.1 > /dev/null
pet@sender:~$ arp -n 192.168.1.1
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.1.1              ether   00:0f:66:5b:2a:3c   C                     eth0

Therefore, 00:0f:66:5b:2a:3c is the MAC address of the receiver.

Step 2: Change boot time kernel command line arguments

For this guide it is assumed that the system under test uses GRUB as its bootloader. By default, GRUB will boot the kernel with command line arguments quiet splash. These need to be changed to capture as much information as possible.

GRUB 2 instructions

Create a backup of /etc/default/grub first: 

pet@sender:~$ sudo cp /etc/default/grub /etc/default/grub.backup

Now, you will need to edit /etc/default/grub as root.

Locate the line GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" and replace this line with GRUB_CMDLINE_LINUX_DEFAULT="debug ignore_loglevel". 

--- /etc/default/grub.backup    2026-10-09 13:31:00.352048642 -0400
+++ /etc/default/grub   2026-10-09 13:31:18.271391144 -0400
@@ -7,7 +7,7 @@
 GRUB_TIMEOUT_STYLE=hidden
 GRUB_TIMEOUT=0
 GRUB_DISTRIBUTOR=`( . /etc/os-release; echo ${NAME:-Ubuntu} ) 2>/dev/null || echo Ubuntu`
-GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
+GRUB_CMDLINE_LINUX_DEFAULT="debug ignore_loglevel"
 GRUB_CMDLINE_LINUX=""

 # If your computer has multiple operating systems installed, then you

Finally, update GRUB with the new config: 

pet@sender:~$ sudo update-grub

Step 3: Initialize netconsole at boot time

Update the list of modules to automatically load.

If /etc/modules-load.d exists

pet@sender:~$ sudo sh -c 'echo netconsole > /etc/modules-load.d/netconsole.conf'

If /etc/modules-load.d does not exist

pet@sender:~$ sudo sh -c 'echo netconsole >> /etc/modules.conf'

To configure netconsole, you will need the IP and MAC address of the receiver, as well as the IP address and network interface of the sender. The receiver's IP and MAC addresses should have been gathered in step 1. To get the network interface and IP address of the sender, run: 

pet@sender:~$ ifconfig -a

Next, fill out the netconsole config by creating/updating /etc/modprobe.d/netconsole.conf: 

pet@sender:~$ sudo sh -c 'echo options netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82 > /etc/modprobe.d/netconsole.conf'

The format for this configuration can be broken down this way: 

netconsole=<LOCAL_PORT>@<SENDER_IP_ADDRESS>/<SENDER_INTERFACE>,<REMOTE_PORT>@<RECEIVER_IP_ADDRESS>/<STEP_1_MAC_ADDRESS>

The local and remote ports are up to you, the example uses 6666 for both.

Step 4: Set up receiver

The receiver now needs to run a process to listen for packets.

Syslog-ng

Install syslog-ng: 

pet@receiver:~$ sudo apt-get install syslog-ng

Edit /etc/syslog-ng/syslog-ng.conf and append these lines to the end of the file: 

source net { udp(ip("0.0.0.0") port(6666)); };
destination netconsole { file("/var/log/$HOST-netconsole.log"); };
log { source(net); destination(netconsole); };

Now restart syslog-ng: 

pet@receiver:~$ sudo /etc/init.d/syslog-ng restart

Continue with Step 5: Capture logs. Once you are done, the logging will be in /var/log/<SENDER_IP_ADDRESS>-netconsole.log.

Netcat

Netcat is an alternative if syslog-ng is not available for whatever reason.

In this example the <RECEIVER_IP_ADDRESS> is 192.168.1.103. (see Step 3: Initialize netconsole at boot time) 

pet@receiver:~$ netcat -l -u 192.168.1.103 6666 | tee ~/netconsole.log

Allow the process to run in the background as you continue with Step 5: Capture logs. Once you are done, press "Control-C" to stop netcat. The messages received will be forwarded to ~/netconsole.log.

You can also try without the IP address specifically listed if you are having issues connecting: 

pet@receiver:~$ netcat -l -u 6666 | tee ~/netconsole.log

On very old versions of netcat, you may see 

UDP listen needs -p arg

which means you need to use netcat-traditional syntax. For example: 

pet@receiver:~$ netcat -l -p 6666 -u | tee ~/netconsole.log

Step 5: Capture logs

Reboot the sender, then execute the command below. 

pet@sender:~$ dmesg | grep netcon
[   21.048406] netconsole: local port 6666
[   21.048410] netconsole: local IP 192.168.1.102
[   21.048411] netconsole: interface eth0
[   21.048413] netconsole: remote port 6666
[   21.048415] netconsole: remote IP 192.168.1.103
[   21.048418] netconsole: remote ethernet address 08:00:46:d4:1d:82
[   21.048421] netconsole: device eth0 not up yet, forcing it
[   22.908106] console [netcon0] enabled
[   22.911536] netconsole: network logging started

Seems to be working! Check your receiver to confirm the kernel messages were received. If logs are appearing in your target file, you can continue to use the sender machine until it crashes.

If you need to avoid rebooting the sender, you can manually load netconsole via: 

sudo modprobe netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82

Substituting the example config for your own.

To test logging before a kernel panic, one option is to insert removable media on the sender, like a CD or USB.

Remove netconsole

Once you've retrieved the information you needed, these are the steps to clean up.

Revert GRUB to its original configuration

GRUB 2 instructions

pet@sender:~$ sudo mv /etc/default/grub.backup /etc/default/grub
pet@sender:~$ sudo update-grub

Remove netconsole itself

If /etc/modules-load.d exists

pet@sender:~$ sudo rm -f /etc/modules-load.d/netconsole.conf
pet@sender:~$ sudo rm -f /etc/modprobe.d/netconsole.conf

If /etc/modules-load.d does not exist

pet@sender:~$ sudo sed -i '/^netconsole/d' /etc/modules
pet@sender:~$ sudo rm -f /etc/modprobe.d/netconsole.conf

See also

Credits

Retrieved from "https://wiki.ubuntu.com/w/index.php?title=Debugging_kernel_panics_remotely_with_netconsole&oldid=1300"