KernelTeamBugPolicies

Filing downstream kernel bug reports

In order to automatically gather and attach necessary system information to a bug report, the preferred method for reporting a Ubuntu kernel bug is run the following command from a terminal window (Applications->Accessories->Terminal):

ubuntu-bug linux

The submitter should provide as much information as possible in the bug description:

  • The majority of kernel bug are hardware specific so be sure to note what hardware/device is being used.
  • Document any known steps to reproduce the bug.
  • Also note whether the bug exists in previous kernel versions of Ubuntu or if it's a regression from previous kernel versions.
  • Finally, it's critical to also make sure to test the latest development Ubuntu kernel version as well as the latest upstream mainline kernel.

If there are any questions about what information to provide, please contact a developer in the #ubuntu-kernel channel on the IRC server for help on how to file a bug.

Adding debug info to a downstream bug report

If you didn't file your Ubuntu kernel bug using ubuntu-bug, please run the following at a terminal, which will automatically gather and attach the requested system information to the bug report:

apport-collect -p linux <bug#>

As a last resort, at a minimum, your bug report should contain the output of the following commands:

uname -a > uname-a.log

cat /proc/version_signature > version.log

dmesg > dmesg.log

sudo lspci -vvnn > lspci-vvnn.log

These four files should be attached separately to the bug report (not pasted into comments or tar and zipped). Please note that dmesg output should be captured as early as possible after bootup to avoid extraneous output.

Kernel sound bugs

If this is a sound related Ubuntu kernel bug, run the following to gather and attach important sound debugging information to the bug:

apport-collect -p alsa-base <bug#>

Problems in capturing information

Bootloader

If the bug occurs during bootup, you can disable the splash screen in one of the following ways:

  • Permanent: Edit /boot/grub/menu.lst and remove splash and quiet kernel parameters to bootloader from the entry for the buggy kernel

  • Temporary: Press 'Escape' key at the 3 second pause by Grub bootloader. Then press 'e' (edit) on the buggy kernel entry, followed by 'e' again on the kernel line. Then remove the splash and quiet keywords and press 'b' to boot.

Capturing OOPs

If the bug report involves a crash, it is hoped that a kernel backtrace (aka OOPS, kernel panic) is available. If the machine does not completely lockup from the crash, the backtrace should be available in the dmesg output. If the crash completely locks the system:

  • Take a look at https://wiki.ubuntu.com/KernelTeam/CrashdumpRecipe.

  • If using linux-crashdump (above) is not successful try and see if any backtrace was logged to /var/log/kern.log.0. Please attach this file if anything was captured.

  • If unable to log the full backtrace, supply a digital photo of the screen to capture the crash. It most important to capture the beginning of the kernel oops or panic.
  • When all fails, try to see if https://wiki.ubuntu.com/KernelTeam/Netconsole can help out.

In X window mode

Sometimes crashes occur in X, and so terminal access is not available (to capture the kernel backtrace). When this occurs, the user should try to recreate the crash at the console (Ctrl+Alt+F1). If this is not possible, then annotate the bug as such.

DIY Debugging Hints

For a complete list of debugging procedures refer to https://wiki.ubuntu.com/DebuggingProcedures

Wiki

Description

DebuggingKernelHibernate

Debugging Hibernate/Resume issues

DebuggingIRQProblems

Debugging interrupt related issues

DebuggingACPI

Debugging ACPI (Advanced Configuration and Power Interface)

DebuggingSystemCrash

Tips for gather information regarding system crashes and lockups

DebuggingSoundProblems

Basic troubleshooting tips for debugging sound problems in Ubuntu

BIOSandUbuntu

Debugging BIOS issues on Ubuntu (DSDT, Reboot, Suspend/Resume)

KernelTeam/LinuxWireless

Debugging wireless device drivers

Bug Triage

Triaging kernel bugs is a day-to-day effort, and can be very time consuming. Luckily, we have a lot of community members willing and able to help with this effort. The kernel team will also begin using a set of Kernel Arsenal scripts to help with the day-to-day triaging efforts. In order to make sure everyone working on kernel bugs follows the same policy, this document will describe how to handle the kernel bug workflow. This will also provide bug reporters with an idea of the life cycle their bug will follow.

Note that beginning with the Karmic development cycle an emphasis is being made to ensure bugs are tested and reported upstream. In relation to this, the Ubuntu kernel team will be transitioning their focus to fixing bugs which have been confirmed to exist upstream or are fixed upstream but exist in the Ubuntu kernel.

New Bugs

Bugs should always come in with a Status of New and an Importance of Undecided. Bugs should automatically have the appropriate debug information attached assuming they were reported using the preferred ubuntu-bug linux method. If this is the case, move the bug to a Confirmed state. If the bug is missing the appropriate debug information, the submitter should be asked to run apport-collect. Note the apport-collect command below only applies to Karmic.

apport-collect -p linux <bug#>

Set the bug to Incomplete. Also tag the bug "needs-kernel-logs". It's also a good idea to subscribe to a bug which you've set to Incomplete. That will ensure you are notified if and when the requested information is provided.

See process-new-bugs.py Kernel Arsenal script [to be linked soon]

Incomplete Bugs

Bugs are typically moved from a New state to an Incomplete state because they are lacking debug information necessary for the Ubuntu Kernel Team to debug the issue. Once all the information has been provided, the bug should be moved from an Incomplete state to a Confirmed state. If the bug was previously tagged "needs-kernel-logs", remove the tag as well once the bug moves to a Confirmed state.

If a bug is in an Incomplete state for more than 120 days and is not updated to provide the requested information, the bug should be expired by setting the status to Invalid and the reason it's being expired should be stated as a comment.

See process-incomplete-bugs.py Kernel Arsenal script [to be linked soon]

Confirmed Bugs

Confirmed bugs should have the appropriate debug information attached. In order for a Confirmed bug to move to a Triaged state, the upstream mainline kernel should be tested as well. This not only helps determine if the bug exists upstream, but also helps determine if a bug might be fixed upstream as well. If the bug exists upstream is also allow additional upstream developers to examine the issue. As mentioned previously, the Ubuntu kernel team will be focusing on bugs which have been confirmed to exist upstream or are fixed upstream but exist in the Ubuntu kernel. If a bug is in a Confirmed state but has not yet tested the upstream mainline kernel, tag the bug "needs-upstream-testing". If a bug has been tested with the upstream kernel, move the bug to a "Triaged" state. If the bug was previously tagged with "needs-upstream-testing", remove the tag once the bug moves to a Triaged state.

See process-confirmed-bugs.py Kernel Arsenal script [to be linked soon]

Triaged Bugs

Once a bug has been tested with the upstream mainline kernel and moved to a Triaged state, the bug should have their Importance set to something other than Undecided. If a bug is Triaged, then there should be enough information to know how important it is. If a bug was tested with the upstream mainline kernel and determined to exist upstream, a bug should also be reported upstream .

Note, you must be a member of the Ubuntu Bug Control team in order to set bugs to Triaged.

In Progress

A bug will move from Triaged to In Progress when a developer has chosen to actively work on the bug. If you are a developer and have set a bug to In Progress, make sure you also assign the bug to yourself. Please keep in mind that a bug marked In Progress for an extended period (greater than a week or two) should be updated periodically with any progress. If you are unable to dedicate time to working on the bug, move the bug back to a Triaged state and unassign yourself from the bug.

Also, do not assign another individual to a bug without their consent first! Doing so just gives bug reporters/subscribers a false sense that someone is actively working on a bug when that may not be the case. Leave it to the discretion of the developer to take ownership of a bug.

Fix Committed

Bugs that are marked Fix Committed are considered fixed by a patch which as been committed to the Ubuntu kernel git repository. This does not mean the fix has been released but it should be expected to be in the next kernel upload. There is no determinate time when a kernel upload will happen, it's up to the discretion of the Ubuntu Kernel Team.

Fix Released

Bugs are moved to a Fix Released state when the fix is readily available in the Ubuntu archive (in the updates pocket). If the Ubuntu kernel developer correctly created the git commit message to include the Launchpad Buglink, the launchpad janitor should automatically move a bug from Fix Committed to Fix Released once the fix is officially available.

Won't Fix

Won't Fix indicates the issue is recognized as a bug but a fix will not be applied, for ex BIOS issues. Some bugs may be marked as Won't Fix for a specific release and Fix Released against another.

Note, you must be a member of the Ubuntu Bug Control team in order to set bugs to Won't Fix.

Invalid

Bugs which are not legitimate bugs are set to Invalid. This can include bugs which failed to provide requested debug information or bugs which are the result of user error.

Tags

Tagging a bug is an easy way to group bugs across packages or within a single package. Tags which are commonly used for kernel bugs are:

The following tags are in use by the kernel team

Subsystem Tag

Owner

Meaning

Links

kernel-therm

apw

A bug related to kernel temperature or fans Kernel/Debugging/HighTemperatures

To Review

kernel-input

ALL

Bugs dealing with input devices

To Review

kernel-uncat

ALL

Bugs that have not been identified for a particular subsystem

To Review

kernel-graphics

sconklin

Bugs that pertain to the graphics subsystem portion of the kernel

To Review

kernel-graphics-vga

ALL

Bugs pertaining to VGA graphics

To Review

kernel-graphics-resolution

ALL

Bugs pertaining to video resolution issues

To Review

kernel-graphics-blankscreen

ALL

Bugs pertaining to blank screen resume/reboot

To Review

kernel-graphics-hdmi

ALL

Bugs pertaining to HDMI video out

To Review

kernel-acpi

cking

Bugs pertaining to acpi related events and systems

To Review

kernel-net

TBC

Bugs dealing with the network stack

To Review

kernel-wifi

ALL

Bugs pertaining to wifi stack/devices not working correctly

To Review

kernel-wifi-needsdriver

ALL

Bugs pertaining to wifi device missing driver

To Review

kernel-wifi-rfkill

ALL

Bugs pertaining to wifi rfkill not working correctly

To Review

kernel-sound

diwic

Bugs related to the sound subsystem

To Review

kernel-sound-microphone

ALL

Bugs related to microphone not working

To Review

kernel-sound-speaker

ALL

Bugs related to speaker not working

To Review

kernel-sound-headphone

ALL

Bugs related to headphone not working

To Review

kernel-sound-hdmi

ALL

Bugs related to hdmi sound out not working

To Review

kernel-fs

apw

Pertaining to the filesystem

To Review

kernel-power

manjo

Bugs related to the power-saving subsystem (includes suspend/resume)

To Review

kernel-core

ALL

Bugs pertaining to internal kernel logic not related to a subsystem

To Review

kernel-media

ALL

Bugs pertaining to SD/MMC/SDHC/CDROM/DVDROM not recognized

To Review

kernel-bluetooth

ALL

Bugs pertaining to bluetooth stack/devices not working correctly

To Review

kernel-hotkey

ALL

Bugs related to hotkey/buttons not working

To Review

Review Tag

Meaning

Links

kernel-needs-review

Needs review by a kernel team member

kernel-reviewed

Has been reviewed by a kernel team member, needs further review by a subject matter expert

kernel-candidate

has been reviewed by a subject matter expert, needs review for inclusion in top list

To Review kernel candidates Top 50 list

Additional used tags:

Tag

Description

kconfig

Used to identify a bug requesting enablement/disablement of specific kernel configuration options.

From Bugs/Tags Tags which are commonly used for kernel bugs are:

Tag

Use case

apport-kerneloops

This Kernel Oops was reported using apport.

bitesize

With regards to the kernel, this includes things like enabling modules and changing kernel config options.

cherry-pick

A kernel bug that has a git commit SHA from the upstream kernel.

hibernate-resume

This bug was triggered by a hibernate/resume failure.

kernel-bug

A "BUG:" message output was noted in the logs but it did not contain an Oops.

kernel-oops

This bug causes a kernel Oops message.

needs-upstream-testing

This bug needs to be tested with the upstream kernel.

kernel-fixed-upstream

This bug is not reproducible with the latest upstream kernel version available that allows the reporter to test it, and the version is higher than the Ubuntu kernel after mapping.

kernel-bug-exists-upstream

This bug is reproducible with the latest upstream kernel version available that allows the reporter to test it, and the version is higher than the Ubuntu kernel after mapping.

suspend-resume

This bug was triggered by a suspend/resume failure.

xorg-needs-kernel-fix

This is an xorg bug which is dependent on a kernel patch.

For more tags see Kernel/Tagging.

Please refer to Bug Tags for more information.

CategoryBugSquad

Please refer to Bug Tags for more information.

Caveats

Sometimes kernel bugs are opened to track security vulnerabilities. These bugs usually contain the word "CVE" either in the title or bug description and will most likely have the ubuntu-security team subscribed to the bug. We should try to avoid spamming these bugs with comments to test the latest kernel to verify if the issue still exists. Whether manually posting to a bug or using python-launchpad-bugs to script comments, please take extra care to not cause more unnecessary traffic for the security team to deal with. Thanks.

Filing upstream kernel bugs

  • If your downstream bug report on Launchpad has been marked Triaged, and a Ubuntu community member asked you to read this, thank you for doing so! By reading and following this completely, you are maximizing the speed with which your bug will be fixed. Please take care to read every step carefully.

First step: Prepare the Kernel.org format information

  • Note the below Kernel.org format was taken directly from upstream.

  • Warning /!\ Please ensure you follow the below format word for word. Just because you tested the latest mainline kernel, may have bisected a kernel regression, or others say they are experiencing the same problem, doesn't mean you should omit anything. Providing this information is vital for a developer to fix your problem, and to maximize the chance of your bug being addressed.

  • Warning /!\ Please take care that when you provide the below information, check that you are booted into the newest available upstream mainline kernel only. Folks have had their bug marked Triaged, sent their e-mail, but had a newer kernel release come out without them realizing this. Hence, if upstream didn't ignore your report altogether, they will ask you to test this in the newest release that just came out and you didn't check for. Failure to do this may have negative unintended consequences. The Ubuntu kernel is the Ubuntu Community's responsibility, the upstream mainline kernel is the Kernel.org Community's responsibility, of which Ubuntu is a part of.


[1.] One line summary of the problem:
Please paste the Bug Description, and remove all references to Ubuntu and Ubuntu's kernels. The reason for this is that the issue you are reporting is an upstream one.

[2.] Full description of the problem/report:
2a. While booted into the newest upstream mainline kernel only, how is the bug reproducible on a keyboard click for click basis?
2b. Is this a regression? If so, after commit bisecting, what is the commit that caused the regression?

[3.] Keywords (i.e., modules, networking, kernel): Please do not put anything here. This is how there is no known keyword system.

[4.] Kernel version (from /proc/version):
While booted into the newest upstream mainline kernel only, please execute the following in a terminal and paste the results:
cat /proc/version

[5.] Output of Oops.. message (if applicable) with symbolic information resolved (see Documentation/oops-tracing.txt)
This is only relevant if you had a kernel oops crash (ex. flashing Caps Lock light). While booted into the newest upstream mainline kernel only, if you have a kernel oops, one may consult http://www.kernel.org/doc/Documentation/oops-tracing.txt . If this is too daunting please ask for help.

[6.] A small shell script or example program which triggers the problem (if possible)
This is for advanced community members. If you feel comfortable enough, please do it. It is not critical if you are unsure on how to do this.

  • Warning /!\ For input problems only
    For reports that would normally be sent to the linux-input mailing list, as requested by the maintainer, please do not send any of the information from section [7.] through [7.7] below.

[7.] Environment
Please execute the following in a terminal, and paste the results:
lsb_release -rd

[7.1.] Software (add the output of the ver_linux script here)
While booted into the newest upstream mainline kernel only, this is found in the directory:
/usr/src/linux-headers-<VERSION>/scripts

where <VERSION> is the version of the kernel you are using, found in the directory /usr/src. You may run the script by changing to the directory via a terminal, and paste the results:
sh ver_linux

[7.2.] Processor information (from /proc/cpuinfo):
While booted into the newest upstream mainline kernel only, execute the following in a terminal, and paste the results:
cat /proc/cpuinfo

[7.3.] Module information (from /proc/modules):
While booted into the newest upstream mainline kernel only, execute the following in a terminal, and paste the results:
cat /proc/modules

[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)
While booted into the newest upstream mainline kernel only, execute the following in a terminal, and paste the results:
cat /proc/ioports
cat /proc/iomem

[7.5.] PCI information ('lspci -vvv' as root)
While booted into the newest upstream mainline kernel only, execute the following via a terminal, and paste the results:
sudo lspci -vvv

[7.6.] SCSI information (from /proc/scsi/scsi)
While booted into the newest upstream mainline kernel only, execute the following via a terminal, and paste the results:
cat /proc/scsi/scsi

[7.7.] Other information that might be relevant to the problem (please look in /proc and include all information that you think to be relevant):
While booted into the newest upstream mainline kernel only, execute the following via a terminal, and paste the results:
ls /proc

[X.] Other notes, patches, fixes, workarounds:

[X.1] Bluetooth problems only - Please execute the following in a terminal and post the results:
sudo cat /sys/kernel/debug/usb/devices

[X.2] Please provide a link to your Launchpad bug report.

[X.3] Any further debugging information from http://www.kernel.org/doc/Documentation/ using the newest upstream mainline kernel available, and all relevant comment information not included in the Bug Description. Doing this makes it easier for kernel developers to review your problem without having to dumpster dive through various links, and increases the chance of your problem being reviewed. If you are unsure about anything asked for, or intend on not following this format, you are reducing the chances of having your problem reviewed. Instead, one would want to get help, until all issues are cleared up.

Second step: Prepare your email to be sent

Now that you have prepared the Kernel.org format information, the next step is formating your email correctly. Specifically:

  1. When sending an email, do so in plain text, not html. Otherwise, it will be flagged as SPAM, and silently discarded without notification to you. For more on this and other reasons your post would be flagged as SPAM, please see here.

  2. Do not send attachments. Instead, post all information in the body of your e-mail.

Third step: With the Kernel.org format send to appropriate maintainers

Reporting Graphics card driver bugs

All other drivers send an E-Mail TO: the maintainers and CC: the maintainer mailing list

  1. Please ensure you send your email TO: all maintainers of the driver in the TO: field, and put the mailing list in the CC: field. A list of both may be found here. The 'M:' is the Maintainer, and the 'L:' is the mailing list. If no list is specified, send an e-mail TO: the maintainers of the relevant driver.

  2. If this is a regression, also put the submitter of the regression commit in the TO: field.
  3. If the author of the driver (which may or may not be the current maintainer) appears still active in development, or the driver isn't listed, one would want to put the author in the TO: field. They may be found via a terminal:

    modinfo DRIVER
    Where DRIVER is the name of the kernel driver.
  4. Please ensure you post in your Launchpad report a URL of your e-mail from the mailing list found from the mailing list archives. It can take a few hours to days for the e-mail to show up to the respective archive.

* Please note that for non-WiFi USB bugs, the USB maintainer(s) do not want anyone to create a Bugzilla report. For more on this, please see this upstream report comment. If you do open an upstream report anyways, it will be immediately closed, and may cause negative unintended consequences.

Fourth step: No response from maintainers

If no response to your email has been received within a week, and the issue is still reproducible with the latest mainline kernel available, then if the issue is related to:

Negative unintended consequences

  • If you contact upstream, please follow the Kernel.org format created by Kernel.org developers. Failure to follow these directions exactly as shown may have the following negative unintended consequences:
    • May likely result in your bug requiring upstream developers to ask unnecessary follow up questions that should have been provided upfront.
    • May likely result in your bug being promptly ignored by the kernel maintainer, sub-maintainer, or community developer(s) responsible to fix your kernel bug.
    • May create friction between you, the Ubuntu Community, and the kernel.org developers, which is not very Ubuntu.


CategoryKernel CategoryBugSquad

KernelTeam/KernelTeamBugPolicies (last edited 2023-05-23 09:46:48 by buo-ren-lin)