Filing downstream kernel bug reports

In order to automatically gather and attach necessary system information to a bug report, the preferred method for reporting a Ubuntu kernel bug is run the following command from a terminal window (Applications->Accessories->Terminal):

ubuntu-bug linux

The submitter should provide as much information as possible in the bug description:

• The majority of kernel bug are hardware specific so be sure to note what hardware/device is being used.
• Document any known steps to reproduce the bug.
• Also note whether the bug exists in previous kernel versions of Ubuntu or if it's a regression from previous kernel versions.

• Finally, it's critical to also make sure to test the latest development Ubuntu kernel version as well as the latest upstream mainline kernel.

If there are any questions about what information to provide, please contact a developer in the #ubuntu-kernel channel on the IRC server for help on how to file a bug.

Adding debug info to a downstream bug report

If you didn't file your Ubuntu kernel bug using ubuntu-bug, please run the following at a terminal, which will automatically gather and attach the requested system information to the bug report:

apport-collect -p linux <bug#>

As a last resort, at a minimum, your bug report should contain the output of the following commands:

uname -a > uname-a.log

cat /proc/version_signature > version.log

dmesg > dmesg.log

sudo lspci -vvnn > lspci-vvnn.log

These four files should be attached separately to the bug report (not pasted into comments or tar and zipped). Please note that dmesg output should be captured as early as possible after bootup to avoid extraneous output.

Kernel sound bugs

If this is a sound related Ubuntu kernel bug, run the following to gather and attach important sound debugging information to the bug:

apport-collect -p alsa-base <bug#>

Problems in capturing information

Bootloader

If the bug occurs during bootup, you can disable the splash screen in one of the following ways:

• Permanent: Edit /boot/grub/menu.lst and remove splash and quiet kernel parameters to bootloader from the entry for the buggy kernel

• Temporary: Press 'Escape' key at the 3 second pause by Grub bootloader. Then press 'e' (edit) on the buggy kernel entry, followed by 'e' again on the kernel line. Then remove the splash and quiet keywords and press 'b' to boot.

Capturing OOPs

If the bug report involves a crash, it is hoped that a kernel backtrace (aka OOPS, kernel panic) is available. If the machine does not completely lockup from the crash, the backtrace should be available in the dmesg output. If the crash completely locks the system:

• Take a look at https://wiki.ubuntu.com/KernelTeam/CrashdumpRecipe.

• If using linux-crashdump (above) is not successful try and see if any backtrace was logged to /var/log/kern.log.0. Please attach this file if anything was captured.

• If unable to log the full backtrace, supply a digital photo of the screen to capture the crash. It most important to capture the beginning of the kernel oops or panic.

• When all fails, try to see if https://wiki.ubuntu.com/KernelTeam/Netconsole can help out.

In X window mode

Sometimes crashes occur in X, and so terminal access is not available (to capture the kernel backtrace). When this occurs, the user should try to recreate the crash at the console (Ctrl+Alt+F1). If this is not possible, then annotate the bug as such.

DIY Debugging Hints

For a complete list of debugging procedures refer to https://wiki.ubuntu.com/DebuggingProcedures

Bug Triage

Triaging kernel bugs is a day-to-day effort, and can be very time consuming. Luckily, we have a lot of community members willing and able to help with this effort. The kernel team will also begin using a set of Kernel Arsenal scripts to help with the day-to-day triaging efforts. In order to make sure everyone working on kernel bugs follows the same policy, this document will describe how to handle the kernel bug workflow. This will also provide bug reporters with an idea of the life cycle their bug will follow.

Note that beginning with the Karmic development cycle an emphasis is being made to ensure bugs are tested and reported upstream. In relation to this, the Ubuntu kernel team will be transitioning their focus to fixing bugs which have been confirmed to exist upstream or are fixed upstream but exist in the Ubuntu kernel.

New Bugs

Bugs should always come in with a Status of New and an Importance of Undecided. Bugs should automatically have the appropriate debug information attached assuming they were reported using the preferred ubuntu-bug linux method. If this is the case, move the bug to a Confirmed state. If the bug is missing the appropriate debug information, the submitter should be asked to run apport-collect. Note the apport-collect command below only applies to Karmic.

apport-collect -p linux <bug#>

Set the bug to Incomplete. Also tag the bug "needs-kernel-logs". It's also a good idea to subscribe to a bug which you've set to Incomplete. That will ensure you are notified if and when the requested information is provided.

See process-new-bugs.py Kernel Arsenal script [to be linked soon]

Incomplete Bugs

Bugs are typically moved from a New state to an Incomplete state because they are lacking debug information necessary for the Ubuntu Kernel Team to debug the issue. Once all the information has been provided, the bug should be moved from an Incomplete state to a Confirmed state. If the bug was previously tagged "needs-kernel-logs", remove the tag as well once the bug moves to a Confirmed state.

If a bug is in an Incomplete state for more than 120 days and is not updated to provide the requested information, the bug should be expired by setting the status to Invalid and the reason it's being expired should be stated as a comment.

See process-incomplete-bugs.py Kernel Arsenal script [to be linked soon]

Confirmed Bugs

Confirmed bugs should have the appropriate debug information attached. In order for a Confirmed bug to move to a Triaged state, the upstream mainline kernel should be tested as well. This not only helps determine if the bug exists upstream, but also helps determine if a bug might be fixed upstream as well. If the bug exists upstream is also allow additional upstream developers to examine the issue. As mentioned previously, the Ubuntu kernel team will be focusing on bugs which have been confirmed to exist upstream or are fixed upstream but exist in the Ubuntu kernel. If a bug is in a Confirmed state but has not yet tested the upstream mainline kernel, tag the bug "needs-upstream-testing". If a bug has been tested with the upstream kernel, move the bug to a "Triaged" state. If the bug was previously tagged with "needs-upstream-testing", remove the tag once the bug moves to a Triaged state.

See process-confirmed-bugs.py Kernel Arsenal script [to be linked soon]

Triaged Bugs

Once a bug has been tested with the upstream mainline kernel and moved to a Triaged state, the bug should have their Importance set to something other than Undecided. If a bug is Triaged, then there should be enough information to know how important it is. If a bug was tested with the upstream mainline kernel and determined to exist upstream, a bug should also be reported upstream .

Note, you must be a member of the Ubuntu Bug Control team in order to set bugs to Triaged.

In Progress

A bug will move from Triaged to In Progress when a developer has chosen to actively work on the bug. If you are a developer and have set a bug to In Progress, make sure you also assign the bug to yourself. Please keep in mind that a bug marked In Progress for an extended period (greater than a week or two) should be updated periodically with any progress. If you are unable to dedicate time to working on the bug, move the bug back to a Triaged state and unassign yourself from the bug.

Also, do not assign another individual to a bug without their consent first! Doing so just gives bug reporters/subscribers a false sense that someone is actively working on a bug when that may not be the case. Leave it to the discretion of the developer to take ownership of a bug.

Fix Committed

Bugs that are marked Fix Committed are considered fixed by a patch which as been committed to the Ubuntu kernel git repository. This does not mean the fix has been released but it should be expected to be in the next kernel upload. There is no determinate time when a kernel upload will happen, it's up to the discretion of the Ubuntu Kernel Team.

Fix Released

Bugs are moved to a Fix Released state when the fix is readily available in the Ubuntu archive (in the updates pocket). If the Ubuntu kernel developer correctly created the git commit message to include the Launchpad Buglink, the launchpad janitor should automatically move a bug from Fix Committed to Fix Released once the fix is officially available.

Won't Fix

Won't Fix indicates the issue is recognized as a bug but a fix will not be applied, for ex BIOS issues. Some bugs may be marked as Won't Fix for a specific release and Fix Released against another.

Note, you must be a member of the Ubuntu Bug Control team in order to set bugs to Won't Fix.

Invalid

Bugs which are not legitimate bugs are set to Invalid. This can include bugs which failed to provide requested debug information or bugs which are the result of user error.

Tags

Tagging a bug is an easy way to group bugs across packages or within a single package. Tags which are commonly used for kernel bugs are:

Please refer to Bug Tags for more information.

Caveats

Sometimes kernel bugs are opened to track security vulnerabilities. These bugs usually contain the word "CVE" either in the title or bug description and will most likely have the ubuntu-security team subscribed to the bug. We should try to avoid spamming these bugs with comments to test the latest kernel to verify if the issue still exists. Whether manually posting to a bug or using python-launchpad-bugs to script comments, please take extra care to not cause more unnecessary traffic for the security team to deal with. Thanks.

Filing upstream kernel bugs

• If your downstream bug report on Launchpad has been marked Triaged, and a Ubuntu community member asked you to read this, thank you for doing so! By reading and following this completely, you are maximizing the speed with which your bug will be fixed. Please take care to read every step carefully.

First step: Prepare the Kernel.org format information

• Note the below Kernel.org format was taken directly from upstream.

• Please ensure you follow the below format word for word. Just because you tested the latest mainline kernel, may have bisected a kernel regression, or others say they are experiencing the same problem, doesn't mean you should omit anything. Providing this information is vital for a developer to fix your problem, and to maximize the chance of your bug being addressed.

• Please take care that when you provide the below information, check that you are booted into the newest available upstream mainline kernel only. Folks have had their bug marked Triaged, sent their e-mail, but had a newer kernel release come out without them realizing this. Hence, if upstream didn't ignore your report altogether, they will ask you to test this in the newest release that just came out and you didn't check for. Failure to do this may have negative unintended consequences. The Ubuntu kernel is the Ubuntu Community's responsibility, the upstream mainline kernel is the Kernel.org Community's responsibility, of which Ubuntu is a part of.

• For input problems only
  For reports that would normally be sent to the linux-input mailing list, as requested by the maintainer, please do not send any of the information from section [7.] through [7.7] below.

Second step: Prepare your email to be sent

Now that you have prepared the Kernel.org format information, the next step is formating your email correctly. Specifically:

1. When sending an email, do so in plain text, not html. Otherwise, it will be flagged as SPAM, and silently discarded without notification to you. For more on this and other reasons your post would be flagged as SPAM, please see here.

2. Do not send attachments. Instead, post all information in the body of your e-mail.

Third step: With the Kernel.org format send to appropriate maintainers

Reporting Graphics card driver bugs

• The following graphics card driver bugs are submitted to the respective bug tracker below:

  • nouveau - http://nouveau.freedesktop.org/wiki/Bugs/

  • Intel

    1. Ensure you have tested drm-intel-nightly. The Ubuntu Kernel team has made this available from here.

    2. While booted into drm-intel-nightly capture dmesg after booting with the drm.debug=14 kernel boot parameter.

    3. After the above is collected, report it to the intel-gfx@lists.freedesktop.org mailing list.

All other drivers send an E-Mail TO: the maintainers and CC: the maintainer mailing list

1. Please ensure you send your email TO: all maintainers of the driver in the TO: field, and put the mailing list in the CC: field. A list of both may be found here. The 'M:' is the Maintainer, and the 'L:' is the mailing list. If no list is specified, send an e-mail TO: the maintainers of the relevant driver.

2. If this is a regression, also put the submitter of the regression commit in the TO: field.

3. If the author of the driver (which may or may not be the current maintainer) appears still active in development, or the driver isn't listed, one would want to put the author in the TO: field. They may be found via a terminal:

   modinfo DRIVER

   Where DRIVER is the name of the kernel driver.

4. Please ensure you post in your Launchpad report a URL of your e-mail from the mailing list found from the mailing list archives. It can take a few hours to days for the e-mail to show up to the respective archive.

* Please note that for non-WiFi USB bugs, the USB maintainer(s) do not want anyone to create a Bugzilla report. For more on this, please see this upstream report comment. If you do open an upstream report anyways, it will be immediately closed, and may cause negative unintended consequences.

Fourth step: No response from maintainers

If no response to your email has been received within a week, and the issue is still reproducible with the latest mainline kernel available, then if the issue is related to:

• Intel graphics driver bugs - Please report to https://01.org/linuxgraphics/documentation/how-report-bugs.

• All others - You would want to send a follow up e-mail to the mailing list with the updated kernel testing results, and relevant verbose debugging information.

Negative unintended consequences

• If you contact upstream, please follow the Kernel.org format created by Kernel.org developers. Failure to follow these directions exactly as shown may have the following negative unintended consequences:
  • May likely result in your bug requiring upstream developers to ask unnecessary follow up questions that should have been provided upfront.
  • May likely result in your bug being promptly ignored by the kernel maintainer, sub-maintainer, or community developer(s) responsible to fix your kernel bug.

  • May create friction between you, the Ubuntu Community, and the kernel.org developers, which is not very Ubuntu.

CategoryKernel CategoryBugSquad