Netconsole
This page is part of the debugging series — pages with debugging details for a variety of Ubuntu packages. |
Contents
Introduction
Sometimes it's hard to capture a kernel panic: You don't know how to reproduce the kernel panic and/or once you have rebooted your system, there are no clues in the logfiles. This is where netconsole might help out. Netconsole is a kernel module that logs kernel printk messages over UDP allowing debugging of problems where disk logging fails. This is just a techno-way of saying that the kernel messages will get send over the network in certain packets (UDP packets).
There are a couple of disadvantages though:
- Since netconsole needs to send it's UDP packets to some other host, you need to setup a "receiver" as well
- Netconsole initializes when modules are loaded into the kernel. This doesn't allow capture of early kernel panics
- The driver for the network card must support polling (netpoll api)
- For best results, a reboot is required
- Only IP networks, UDP packets and ethernet devices are supported
Setup netconsole
This document will guide you through the steps to setup netconsole for Ubuntu Linux.
There are 5 steps to set it all up! Good luck!
Step 1: Determine remote mac-address
We need to know a mac-address where the UDP packets must get send to (also known as the "receiver"). This "receiver" can or can't be in the same subnet:
When "receiver" is in same subnet
In this example I would like to send the UDP packets to 192.168.1.103 (which is in the same subnet as the sender is).
pet@sender:~$ ping -c 1 192.168.1.103 > /dev/null pet@sender:~$ arp -n 192.168.1.103 Address HWtype HWaddress Flags Mask Iface 192.168.1.103 ether 08:00:46:d4:1d:82 C eth0
In this example 08:00:46:d4:1d:82 is the macaddress we need.
When "receiver" isn't in the same subnet
Okay, when that's the case, you need to determine the default gateway first:
pet@sender:~$ netstat -rn | grep ^0.0.0.0 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
In this case, that's 192.168.1.1.
Now, we need to figure out which macaddress our default gateway has:
pet@sender:~$ ping -c 1 192.168.1.1 > /dev/null pet@sender:~$ arp -n 192.168.1.1 Address HWtype HWaddress Flags Mask Iface 192.168.1.1 ether 00:0f:66:5b:2a:3c C eth0
In this example 00:0f:66:5b:2a:3c is the macaddress we need.
Step 2: Change kernel options at boot time
Okay, let's assume you use grub as your bootloader. In this case grub will boot the kernel with (at least) the "quiet splash" options by default. We don't want that.
Just to be sure, create a backup of /boot/grub/menu.lst first:
pet@sender:~$ sudo cp /boot/grub/menu.lst /root/menu.lst.backup
Now, open your favorite editor to edit /boot/grub/menu.lst
- For vi:
sudo vi /boot/grub/menu.lst
- For gedit (to edit this file within Gnome):
gksudo gedit /boot/grub/menu.lst
Locate the line that starts with "# defoptions=quiet splash" (don't get mislead by the fact the line starts with a "#") and replace this line with "# defoptions=debug".
Tell grub to update accordingly:
pet@sender:~$ sudo update-grub
Step 3: Initialize netconsole at boot time
Okay, that went well. Three more steps to go!
Now we need to know the:
- ipaddress and interface of the "sender" (use the following commands to determine these)
ifconfig -a
ipaddress and macaddress of the "receiver" (these were already gathered in #Step 1: Determine remote mac-address)
In the following example, the part that starts with "netconsole=" is setup like this:
netconsole=<LOCAL_PORT>@<SENDER_IP_ADDRESS>/<SENDER_INTERFACE>,<REMOTE_PORT>@<RECEIVER_IP_ADDRESS>/<STEP_1_MAC_ADDRESS>
In this example, I took 6666 for both <LOCAL_PORT> and <REMOTE_PORT>.
$ sudo sh -c 'echo netconsole >> /etc/modules' $ sudo sh -c 'echo options netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82 > /etc/modprobe.d/netconsole.conf'
Step 4: Setup receiver
There are several ways to retrieve the netconsole packets:
Syslog-ng (Linux)
Easiest way probably is to boot the receiver with an Ubuntu LiveCD. Setup networking and gain shell access.
pet@receiver:~$ sudo apt-get install syslog-ng
Edit /etc/syslog-ng/syslog-ng.conf with your favorite editor and append these lines at the very end of the file:
source net { udp(ip("0.0.0.0") port(6666)); }; destination netconsole { file("/var/log/$HOST-netconsole.log"); }; log { source(net); destination(netconsole); };
Now restart syslog-ng:
pet@receiver:~$ sudo /etc/init.d/syslog-ng restart
Continue with #Step 5: Check if it works. Once you are done, the logging will be in /var/log/<SENDER_IP_ADDRESS>-netconsole.log.
Netcat (Linux)
If you have the luxury of having a Linux machine as "receiver", you just might want to try this:
In this example 192.168.1.103 is the same as "<RECEIVER_IP_ADDRESS>" (see #Step 3: Initialize netconsole at boot time)
pet@receiver:~$ netcat -l -p 6666 -u 192.168.1.103 | tee ~/netconsole.log
Just let it sit there. Continue with #Step 5: Check if it works. Once you are done, press "Control-C". The messages you want are in ~/netconsole.log.
If this doesn't work, also try without the IP address specifically listed:
pet@receiver:~$ netcat -l -p 6666 -u | tee ~/netconsole.log
Netcat (Windows)
There's a netcat for Windows available here or here.
Unpack it somewhere (i.e. C:\Users\Pet\Desktop\nc\)
Open a dosprompt (Start -> Run -> cmd). In this example 192.168.1.103 is the same as <RECEIVER_IP_ADDRESS>.
cd C:\Users\Pet\Desktop\nc\ nc -u -l -p 6666 192.168.1.103 > netconsole.txt
Just let it sit there. Continue with #Step 5: Check if it works. Once you are done, press "Control-C". The messages you want are in netconsole.txt.
Syslog (Windows)
There's an open source syslog tool here. In the version currently available (2009, Jan), there is no way to change the port syslog listens on. You'll have to change the <REMOTE_PORT> to port 514 (which is the default syslog port).
Step 5: Check if it works
Reboot the "sender". Once rebooted execute the command below.
$ dmesg | grep netcon [ 21.048406] netconsole: local port 6666 [ 21.048410] netconsole: local IP 192.168.1.102 [ 21.048411] netconsole: interface eth0 [ 21.048413] netconsole: remote port 6666 [ 21.048415] netconsole: remote IP 192.168.1.103 [ 21.048418] netconsole: remote ethernet address 08:00:46:d4:1d:82 [ 21.048421] netconsole: device eth0 not up yet, forcing it [ 22.908106] console [netcon0] enabled [ 22.911536] netconsole: network logging started
Seems to be working! Now check your "receiver" to see whether the kernel messages were received! If that's the case, continue to use the "sender" until it crashes, hopefully the "receiver" receives some useful information about the crash.
Remove netconsole
Once you retrieved the information you needed, .. we need to clean things up again:
pet@sender:~$ sudo mv /root/menu.lst.backup /boot/grub/menu.lst pet@sender:~$ sudo update-grub pet@sender:~$ sudo sed -i '/^netconsole/d' /etc/modules pet@sender:~$ sudo rm -f /etc/modprobe.d/netconsole.conf
Note from author
Do you have comments on the content above? Something incomplete or incorrect? Please don't hesitate to drop me a few lines.
I would like to say thanks to Peter Veerman and Leann Ogasawara for reviewing this document.
See also
https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Problems%20in%20capturing%20information