Ubuntu Wiki

Debugging kernels with USB earlyprintk

Using this technique one can capture early kernel messages over USB. This uses a special debug port on the targe machine that is to be debugged. Note that this is NOT the same as dumping debug state over a USB serial dongle.

This requires two computers with USB 2.0 ports, a special USB debug 'key' and two USB cables, connected together as follows:

target <---- USB cable ---> [ USB debug key ] <---- USB cable ----> capture client

The USB debug key available from the plextech Net20DC or Ajays Net20DC. It must be connected directly to the target without using a USB hub in between.

The "target" machine is the machine to be debugged. It must have the special USB debug port capability, one can check for this as follows:

sudo lspci -vv

..and you will see:

00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) (prog-if 20 [EHCI])
        Subsystem: Lenovo Device 21da
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at 9252a000 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Debug port: BAR=1 offset=00a0
        Capabilities: [98] PCI Advanced Features
                AFCap: TP+ FLR+
                AFCtrl: FLR-
                AFStatus: TP+
        Kernel driver in use: ehci_hcd

If you see a Capabilities field with a "Debug port" then one can debug over USB.

The EHCI debug controller is bound to a physical USB port and is not software selectable. One has to find the physical USB port by experimentally working through each USB port and rebooting the target machine and observing debug messages being received on the capture client. If you are unlucky you may find that the manufacturer has not connected up the USB debug port to an external USB connector, so beware that this debug technical is not guaranteed to work on all machines.

If you can't get any debug messages then swap the USB debug key around and try again - the devices is powered from USB and the 5V power needs to be connected to the capture client so that it does not get rebooted when the target is rebooted.

Setup

The target kernel needs to be configured with CONFIG_EARLY_PRINTK_DBGP=y which enables the early printk debug. For Ubuntu Quantal 12.10 this is already enabled by default.

The target machine requires one to turn on the early printk debug, to do so, edit /etc/default/grub and change

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

to

GRUB_CMDLINE_LINUX_DEFAULT="earlyprintk=dbgp,keep"

..for USB debug port 0. If you are using a different debug port, one can select the Nth port (starting at zero) using "earlyprintk=dbgpN,keep"; for example port 1, use:

GRUB_CMDLINE_LINUX_DEFAULT="earlyprintk=dbgp1,keep"

..and then run:

sudo update-grub

Finally, we want to remove the modemmanager on the target machine because it will start writing down the USB tty once userspace starts up, so remove it using:

sudo apt-get remove modemmanager

The capture client needs to be configured with CONFIG_USB_SERIAL_DEBUG=y (which is enabled by default for Ubuntu). Once the USB cable is plugged into the capture client one will get a /dev/ttyUSBx device appearing and one can then copy out the messages using a terminal emulator, such as minicom or just dump them using cat, e.g.

cat /dev/ttyUSB0

Testing

You can quickly test to see if your configuration works by writing message to the kernel log on the target machine and seeing if it is echo'd out on the capture client, for example:

echo "Hello World" | sudo tee /dev/kmsg

If this works, you are ready to reboot the target machine and hopefully you will see kernel messages appear on the capture client.

If it doesn't work, try different USB ports on the target machine, or swap the USB debug key around, or select a different port name using "earlyprintk=dbgpN", where N is the port number.