This page is part of the debugging series — pages with debugging details for a variety of Ubuntu packages. |
Contents
Introduction
Sometimes it's hard to capture a kernel panic: You don't know how to reproduce the kernel panic and/or once you have rebooted your system, there are no clues in the logfiles. This is where netconsole might help out. Netconsole is a kernel module that logs kernel printk messages over UDP allowing debugging of problems where disk logging fails. This is just a techno-way of saying that the kernel messages will get send over the network in certain packets (UDP packets).
There are a couple of disadvantages though:
- Since netconsole needs to send it's UDP packets to some other host, you need to setup a "receiver" as well
- Netconsole initializes when modules are loaded into the kernel. This doesn't allow capture of early kernel panics - for example when booting
- The driver for the network card must support polling (netpoll api)
- Only IP networks, UDP packets and ethernet devices are supported
Setup netconsole
This document will guide you through the steps to setup netconsole for Ubuntu {*}.
There are 5 steps to set it all up! Good luck! 
Step 1: Determine remote mac address
We need to know a mac address where the UDP packets will get sent to (also known as the "receiver"). This "receiver" can or can't be in the same subnet:
When "receiver" is in same subnet
In this example I would like to send the UDP packets to 192.168.1.103 (which is in the same subnet as the sender is).
pet@sender:~$ ping -c 1 192.168.1.103 > /dev/null pet@sender:~$ arp -n 192.168.1.103 Address HWtype HWaddress Flags Mask Iface 192.168.1.103 ether 08:00:46:d4:1d:82 C eth0
In this example 08:00:46:d4:1d:82 is the mac address we need.
When "receiver" isn't in the same subnet
Okay, when that's the case, you need to determine the default gateway first:
pet@sender:~$ netstat -rn | grep ^0.0.0.0 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
In this case, that's 192.168.1.1.
Now, we need to figure out which mac address our default gateway has:
pet@sender:~$ ping -c 1 192.168.1.1 > /dev/null pet@sender:~$ arp -n 192.168.1.1 Address HWtype HWaddress Flags Mask Iface 192.168.1.1 ether 00:0f:66:5b:2a:3c C eth0
In this example 00:0f:66:5b:2a:3c is the mac address we need.
Step 2: Change kernel options at boot time
Okay, let's assume you use grub as your bootloader. In this case grub will boot the kernel with (at least) the "quiet splash" options by default. We don't want that.
Grub 0.97: if you are using Ubuntu 9.04 or older
Just to be sure, create a backup of /boot/grub/menu.lst first:
pet@sender:~$ sudo cp /boot/grub/menu.lst /root/menu.lst.backup
Now, open your favorite editor to edit /boot/grub/menu.lst.
- For vi:
sudo vi /boot/grub/menu.lst
- For gedit (to edit this file within Gnome):
gksudo gedit /boot/grub/menu.lst
Locate the line that starts with "# defoptions=quiet splash" (don't get mislead by the fact the line starts with a "#") and replace this line with "# defoptions=debug ignore_loglevel".
Tell grub to update accordingly:
pet@sender:~$ sudo update-grub
Grub 2: if you are using Ubuntu 9.10 or newer
Just to be sure, create a backup of /etc/default/grub first:
pet@sender:~$ sudo cp /etc/default/grub /etc/default/grub.backup
Now, open your favorite editor to edit /etc/default/grub.
- For vi:
sudo vi /etc/default/grub
- For gedit (to edit this file within Gnome):
gksudo gedit /etc/default/grub
Locate the line GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" and replace this line with GRUB_CMDLINE_LINUX_DEFAULT="debug ignore_loglevel".
Tell grub to update accordingly:
pet@sender:~$ sudo update-grub
Step 3: Initialize netconsole at boot time
Okay, that went well. Three more steps to go!
Now we need to know the:
- ip address and interface of the "sender" (use the following commands to determine these)
ifconfig -a
ip address and mac address of the "receiver" (these were already gathered in #Step 1: Determine remote mac address)
In the following example, the part that starts with "netconsole=" is setup like this:
netconsole=<LOCAL_PORT>@<SENDER_IP_ADDRESS>/<SENDER_INTERFACE>,<REMOTE_PORT>@<RECEIVER_IP_ADDRESS>/<STEP_1_MAC_ADDRESS>
In this example, I use 6666 for both <LOCAL_PORT> and <REMOTE_PORT>.
$ sudo sh -c 'echo netconsole >> /etc/modules' $ sudo sh -c 'echo options netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82 > /etc/modprobe.d/netconsole.conf'
Step 4: Setup receiver
There are several ways to retrieve the netconsole packets:
Syslog-ng (Linux)
Easiest way probably is to boot the receiver with an Ubuntu LiveCD. Setup networking and gain shell access.
pet@receiver:~$ sudo apt-get install syslog-ng
Edit /etc/syslog-ng/syslog-ng.conf with your favorite editor and append these lines at the very end of the file:
source net { udp(ip("0.0.0.0") port(6666)); };
destination netconsole { file("/var/log/$HOST-netconsole.log"); };
log { source(net); destination(netconsole); };Now restart syslog-ng:
pet@receiver:~$ sudo /etc/init.d/syslog-ng restart
Continue with #Step 5: Check if it works. Once you are done, the logging will be in /var/log/<SENDER_IP_ADDRESS>-netconsole.log.
Netcat (Linux)
If you have the luxury of having a Linux machine as "receiver", you just might want to try this:
In this example 192.168.1.103 is the same as "<RECEIVER_IP_ADDRESS>" (see #Step 3: Initialize netconsole at boot time)
pet@receiver:~$ netcat -l -u 192.168.1.103 6666 | tee ~/netconsole.log
Just let it sit there. Continue with #Step 5: Check if it works. Once you are done, press "Control-C". The messages you want are in ~/netconsole.log.
If this doesn't work, also try without the IP address specifically listed:
pet@receiver:~$ netcat -l -u 6666 | tee ~/netconsole.log
Netcat (Linux, alternative)
Some versions of Linux, such as Ubuntu Hardy (8.04), use a different version of Netcat that uses a different syntax. If you use a Linux receiver that has a different version of Netcat, it will print the following error message when you try the commands above:
UDP listen needs -p arg
In this case, you need to issue the following command:
pet@receiver:~$ netcat -l -p 6666 -u | tee ~/netconsole.log
Netcat (Mac OS-X)
Unsurprisingly, as OS-X is a BSD UNIX derivative, Netcat is available out of the box on this platform with the minor difference from Linux that the command is actually called nc rather than netcat.
In this example 192.168.1.103 is the same as "<RECEIVER_IP_ADDRESS>" (see #Step 3: Initialize netconsole at boot time)
receiver:~ pet$ nc -l -u 192.168.1.103 6666 | tee ~/netconsole.log
Just let it sit there. Continue with #Step 5: Check if it works. Once you are done, press "Control-C". The messages you want are in ~/netconsole.log.
If this doesn't work, also try without the IP address specifically listed:
receiver:~ pet$ nc -l -u 6666 | tee ~/netconsole.log
Netcat (Windows)
There's a netcat for Windows available here or here.
Unpack it somewhere (i.e. C:\Users\Pet\Desktop\nc\)
Open a dosprompt (Start -> Run -> cmd). In this example 192.168.1.103 is the same as <RECEIVER_IP_ADDRESS>.
cd C:\Users\Pet\Desktop\nc\ nc -u -l -p 6666 192.168.1.103 > netconsole.txt
Just let it sit there. Continue with #Step 5: Check if it works. Once you are done, press "Control-C". The messages you want are in netconsole.txt.
Syslog (Windows)
There's an open source syslog tool here. In the version currently available (2009, Jan), there is no way to change the port syslog listens on. You'll have to change the <REMOTE_PORT> to port 514 (which is the default syslog port).
Step 5: Check if it works
Reboot the "sender". Once rebooted execute the command below.
$ dmesg | grep netcon [ 21.048406] netconsole: local port 6666 [ 21.048410] netconsole: local IP 192.168.1.102 [ 21.048411] netconsole: interface eth0 [ 21.048413] netconsole: remote port 6666 [ 21.048415] netconsole: remote IP 192.168.1.103 [ 21.048418] netconsole: remote ethernet address 08:00:46:d4:1d:82 [ 21.048421] netconsole: device eth0 not up yet, forcing it [ 22.908106] console [netcon0] enabled [ 22.911536] netconsole: network logging started
Seems to be working! 
Now check your "receiver" to see whether the kernel messages were received! If that's the case, continue to use the "sender" until it crashes, hopefully the "receiver" receives some useful information about the crash.
You don't actually need to reboot the sender if you don't want to. You can manually load netconsole via:
$ sudo modprobe netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82
One way to test whether logging is working correctly or not is to insert some removable media like a CD, DVD or usb stick.
Remove netconsole
Once you retrieved the information you needed, we need to clean things up again.
Revert Grub to its original configuration
Grub 0.97: if you are using Ubuntu 9.04 or older
pet@sender:~$ sudo mv /root/menu.lst.backup /boot/grub/menu.lst pet@sender:~$ sudo update-grub
Grub 2: if you are using Ubuntu 9.10 or newer
pet@sender:~$ sudo mv /etc/default/grub.backup /etc/default/grub pet@sender:~$ sudo update-grub
Remove netconsole itself
pet@sender:~$ sudo sed -i '/^netconsole/d' /etc/modules pet@sender:~$ sudo rm -f /etc/modprobe.d/netconsole.conf
Note from author
Do you have comments on the content above? Something incomplete or incorrect? Please don't hesitate to drop me a few lines.
I would like to say thanks to Peter Veerman and Leann Ogasawara for reviewing this document.
A big thanks to Bruno Girin for adding the grub-2 instructions
See also
https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Problems%20in%20capturing%20information