KernelTeamBugPolicies

Filing downstream kernel bug reports

In order to automatically gather and attach necessary system information to a bug report, the preferred method for reporting a Ubuntu kernel bug is run the following command from a terminal window (Applications->Accessories->Terminal):

ubuntu-bug linux

The submitter should provide as much information as possible in the bug description:

  • The majority of kernel bug are hardware specific so be sure to note what hardware/device is being used.
  • Document any known steps to reproduce the bug.
  • Also note whether the bug exists in previous kernel versions of Ubuntu or if it's a regression from previous kernel versions.
  • Finally, it's critical to also make sure to test the latest development Ubuntu kernel version as well as the latest upstream mainline kernel.

If there are any questions about what information to provide, please contact a developer in the #ubuntu-kernel channel on the FreeNode IRC server for help on how to file a bug.

Adding debug info to a downstream bug report

If you didn't file your Ubuntu kernel bug using ubuntu-bug, please run the following at a terminal, which will automatically gather and attach the requested system information to the bug report:

apport-collect -p linux <bug#>

As a last resort, at a minimum, your bug report should contain the output of the following commands:

uname -a > uname-a.log

cat /proc/version_signature > version.log

dmesg > dmesg.log

sudo lspci -vvnn > lspci-vvnn.log

These four files should be attached separately to the bug report (not pasted into comments or tar and zipped). Please note that dmesg output should be captured as early as possible after bootup to avoid extraneous output.

Kernel sound bugs

If this is a sound related Ubuntu kernel bug, run the following to gather and attach important sound debugging information to the bug:

apport-collect -p alsa-base <bug#>

Problems in capturing information

Bootloader

If the bug occurs during bootup, you can disable the splash screen in one of the following ways:

  • Permanent: Edit /boot/grub/menu.lst and remove splash and quiet kernel parameters to bootloader from the entry for the buggy kernel

  • Temporary: Press 'Escape' key at the 3 second pause by Grub bootloader. Then press 'e' (edit) on the buggy kernel entry, followed by 'e' again on the kernel line. Then remove the splash and quiet keywords and press 'b' to boot.

Capturing OOPs

If the bug report involves a crash, it is hoped that a kernel backtrace (aka OOPS, kernel panic) is available. If the machine does not completely lockup from the crash, the backtrace should be available in the dmesg output. If the crash completely locks the system:

  • Take a look at https://wiki.ubuntu.com/KernelTeam/CrashdumpRecipe.

  • If using linux-crashdump (above) is not successful try and see if any backtrace was logged to /var/log/kern.log.0. Please attach this file if anything was captured.

  • If unable to log the full backtrace, supply a digital photo of the screen to capture the crash. It most important to capture the beginning of the kernel oops or panic.
  • When all fails, try to see if https://wiki.ubuntu.com/KernelTeam/Netconsole can help out.

In X window mode

Sometimes crashes occur in X, and so terminal access is not available (to capture the kernel backtrace). When this occurs, the user should try to recreate the crash at the console (Ctrl+Alt+F1). If this is not possible, then annotate the bug as such.

DIY Debugging Hints

For a complete list of debugging procedures refer to https://wiki.ubuntu.com/DebuggingProcedures

Wiki

Description

DebuggingKernelHibernate

Debugging Hibernate/Resume issues

DebuggingIRQProblems

Debugging interrupt related issues

DebuggingACPI

Debugging ACPI (Advanced Configuration and Power Interface)

DebuggingSystemCrash

Tips for gather information regarding system crashes and lockups

DebuggingSoundProblems

Basic troubleshooting tips for debugging sound problems in Ubuntu

BIOSandUbuntu

Debugging BIOS issues on Ubuntu (DSDT, Reboot, Suspend/Resume)

|| KernelTeam/LinuxWireless || Debugging wireless device drivers ||1234541

Bug Triage

Triaging kernel bugs is a day-to-day effort, and can be very time consuming. Luckily, we have a lot of community members willing and able to help with this effort. The kernel team will also begin using a set of Kernel Arsenal scripts to help with the day-to-day triaging efforts. In order to make sure everyone working on kernel bugs follows the same policy, this document will describe how to handle the kernel bug workflow. This will also provide bug reporters with an idea of the life cycle their bug will follow.

Note that beginning with the Karmic development cycle an emphasis is being made to ensure bugs are tested and reported upstream. In relation to this, the Ubuntu kernel team will be transitioning their focus to fixing bugs which have been confirmed to exist upstream or are fixed upstream but exist in the Ubuntu kernel.

New Bugs

Bugs should always come in with a Status of New and an Importance of Undecided. Bugs should automatically have the appropriate debug information attached assuming they were reported using the preferred ubuntu-bug linux method. If this is the case, move the bug to a Confirmed state. If the bug is missing the appropriate debug information, the submitter should be asked to run apport-collect. Note the apport-collect command below only applies to Karmic.

apport-collect -p linux <bug#>

In Jaunty, use

apport-collect -p linux-image-`uname -r` <bug #>

Set the bug to Incomplete. Also tag the bug "needs-kernel-logs". It's also a good idea to subscribe to a bug which you've set to Incomplete. That will ensure you are notified if and when the requested information is provided.

See process-new-bugs.py Kernel Arsenal script [to be linked soon]

Incomplete Bugs

Bugs are typically moved from a New state to an Incomplete state because they are lacking debug information necessary for the Ubuntu Kernel Team to debug the issue. Once all the information has been provided, the bug should be moved from an Incomplete state to a Confirmed state. If the bug was previously tagged "needs-kernel-logs", remove the tag as well once the bug moves to a Confirmed state.

If a bug is in an Incomplete state for more than 120 days and is not updated to provide the requested information, the bug should be expired by setting the status to Invalid and the reason it's being expired should be stated as a comment.

See process-incomplete-bugs.py Kernel Arsenal script [to be linked soon]

Confirmed Bugs

Confirmed bugs should have the appropriate debug information attached. In order for a Confirmed bug to move to a Triaged state, the upstream mainline kernel should be tested as well. This not only helps determine if the bug exists upstream, but also helps determine if a bug might be fixed upstream as well. If the bug exists upstream is also allow additional upstream developers to examine the issue. As mentioned previously, the Ubuntu kernel team will be focusing on bugs which have been confirmed to exist upstream or are fixed upstream but exist in the Ubuntu kernel. If a bug is in a Confirmed state but has not yet tested the upstream mainline kernel, tag the bug "needs-upstream-testing". If a bug has been tested with the upstream kernel, move the bug to a "Triaged" state. If the bug was previously tagged with "needs-upstream-testing", remove the tag once the bug moves to a Triaged state.

See process-confirmed-bugs.py Kernel Arsenal script [to be linked soon]

Triaged Bugs

Once a bug has been tested with the upstream mainline kernel and moved to a Triaged state, the bug should have their Importance set to something other than Undecided. If a bug is Triaged, then there should be enough information to know how important it is. If a bug was tested with the upstream mainline kernel and determined to exist upstream, a bug should also be reported upstream .

Note, you must be a member of the Ubuntu Bug Control team in order to set bugs to Triaged.

In Progress

A bug will move from Triaged to In Progress when a developer has chosen to actively work on the bug. If you are a developer and have set a bug to In Progress, make sure you also assign the bug to yourself. Please keep in mind that a bug marked In Progress for an extended period (greater than a week or two) should be updated periodically with any progress. If you are unable to dedicate time to working on the bug, move the bug back to a Triaged state and unassign yourself from the bug.

Also, do not assign another individual to a bug without their consent first! Doing so just gives bug reporters/subscribers a false sense that someone is actively working on a bug when that may not be the case. Leave it to the discretion of the developer to take ownership of a bug.

Fix Committed

Bugs that are marked Fix Committed are considered fixed by a patch which as been committed to the Ubuntu kernel git repository. This does not mean the fix has been released but it should be expected to be in the next kernel upload. There is no determinate time when a kernel upload will happen, it's up to the discretion of the Ubuntu Kernel Team.

Fix Released

Bugs are moved to a Fix Released state when the fix is readily available in the Ubuntu archive (in the updates pocket). If the Ubuntu kernel developer correctly created the git commit message to include the Launchpad Buglink, the launchpad janitor should automatically move a bug from Fix Committed to Fix Released once the fix is officially available.

Won't Fix

Won't Fix indicates the issue is recognized as a bug but a fix will not be applied, for ex BIOS issues. Some bugs may be marked as Won't Fix for a specific release and Fix Released against another.

Note, you must be a member of the Ubuntu Bug Control team in order to set bugs to Won't Fix.

Invalid

Bugs which are not legitimate bugs are set to Invalid. This can include bugs which failed to provide requested debug information or bugs which are the result of user error.

Tags

Tagging a bug is an easy way to group bugs across packages or within a single package. Tags which are commonly used for kernel bugs are:

The following tags are in use by the kernel team

Subsystem Tag

Owner

Meaning

Links

kernel-therm

apw

A bug related to kernel temperature or fans Kernel/Debugging/HighTemperatures

To Review

kernel-input

ALL

Bugs dealing with input devices

To Review

kernel-uncat

ALL

Bugs that have not been identified for a particular subsystem

To Review

kernel-graphics

sconklin

Bugs that pertain to the graphics subsystem portion of the kernel

To Review

kernel-graphics-vga

ALL

Bugs pertaining to VGA graphics

To Review

kernel-graphics-resolution

ALL

Bugs pertaining to video resolution issues

To Review

kernel-graphics-blankscreen

ALL

Bugs pertaining to blank screen resume/reboot

To Review

kernel-graphics-hdmi

ALL

Bugs pertaining to HDMI video out

To Review

kernel-acpi

cking

Bugs pertaining to acpi related events and systems

To Review

kernel-net

TBC

Bugs dealing with the network stack

To Review

kernel-wifi

ALL

Bugs pertaining to wifi stack/devices not working correctly

To Review

kernel-wifi-needsdriver

ALL

Bugs pertaining to wifi device missing driver

To Review

kernel-wifi-rfkill

ALL

Bugs pertaining to wifi rfkill not working correctly

To Review

kernel-sound

diwic

Bugs related to the sound subsystem

To Review

kernel-sound-microphone

ALL

Bugs related to microphone not working

To Review

kernel-sound-speaker

ALL

Bugs related to speaker not working

To Review

kernel-sound-headphone

ALL

Bugs related to headphone not working

To Review

kernel-sound-hdmi

ALL

Bugs related to hdmi sound out not working

To Review

kernel-fs

apw

Pertaining to the filesystem

To Review

kernel-power

manjo

Bugs related to the power-saving subsystem (includes suspend/resume)

To Review

kernel-core

ALL

Bugs pertaining to internal kernel logic not related to a subsystem

To Review

kernel-media

ALL

Bugs pertaining to SD/MMC/SDHC/CDROM/DVDROM not recognized

To Review

kernel-bluetooth

ALL

Bugs pertaining to bluetooth stack/devices not working correctly

To Review

kernel-hotkey

ALL

Bugs related to hotkey/buttons not working

To Review

Review Tag

Meaning

Links

kernel-needs-review

Needs review by a kernel team member

kernel-reviewed

Has been reviewed by a kernel team member, needs further review by a subject matter expert

kernel-candidate

has been reviewed by a subject matter expert, needs review for inclusion in top list

To Review kernel candidates Top 50 list

Additional used tags:

Tag

Description

kconfig

Used to identify a bug requesting enablement/disablement of specific kernel configuration options.

From Bugs/Tags Tags which are commonly used for kernel bugs are:

Tag

Use case

`apport-kerneloops`

This Kernel Oops was reported using apport.

`bitesize`

With regards to the kernel, this includes things like enabling modules and changing kernel config options

`cherry-pick`

A kernel bug that has a git commit SHA from the upstream kernel

`hibernate resume`

This bug was triggered by a hibernate/resume failure

`kernel-bug`

A "BUG:" message output was noted in the logs but it did not contain an Oops

`kernel-oops`

This bug causes a kernel Oops message

`needs-upstream-testing`

This bug needs to be tested with the upstream kernel - https://wiki.ubuntu.com/KernelMainlineBuilds

`suspend resume`

This bug was triggered by a suspend/resume failure

`xorg-needs-kernel-fix`

This is an xorg bug which is dependent on a kernel patch

For more tags see Kernel/Tagging.

Please refer to Bug Tags for more information.

CategoryBugSquad

Please refer to Bug Tags for more information.

Caveats

Sometimes kernel bugs are opened to track security vulnerabilities. These bugs usually contain the word "CVE" either in the title or bug description and will most likely have the ubuntu-security team subscribed to the bug. We should try to avoid spamming these bugs with comments to test the latest kernel to verify if the issue still exists. Whether manually posting to a bug or using python-launchpad-bugs to script comments, please take extra care to not cause more unnecessary traffic for the security team to deal with. Thanks.

Filing upstream kernel bugs

  • If your downstream bug report on Launchpad has been marked Triaged, and a Ubuntu community member asked you to read this, thank you for doing so! By reading these policies completely, you are maximizing the speed with which your bug will be fixed. Please take care to read every step carefully.

First step: Prepare the Kernel.org format information

  • Note the below Kernel.org format was taken directly from http://kernel.org/pub/linux/docs/lkml/reporting-bugs.html.

  • Warning /!\ Please ensure you follow the below format word for word. Just because you tested the latest mainline kernel, may have bisected a kernel regression, or others say they are experiencing the same problem, doesn't mean you should omit anything. Providing this information is vital for a developer to fix your problem, and to maximize the chance of your bug being addressed.

  • Warning /!\ Please take care that when you provide the below information, you should be booted into the newest available upstream mainline kernel only. Failure to do this will have negative unintended consequences. The Ubuntu kernel is the Ubuntu Community's responsibility, the upstream mainline kernel is the Kernel.org Community's responsibility, of which Ubuntu is a part of.


[1.] One line summary of the problem:
Please paste the Launchpad bug summary, and remove all references to Ubuntu and Ubuntu's kernels. The reason this is being reported upstream is because this is not just a downstream (Ubuntu) issue, but both an up and down one.

[2.] Full description of the problem/report:
While booted into the newest mainline kernel only describe how the bug is reproducible in the latest mainline kernel only. If this is a regression, please note the specific commit.

[3.] Keywords (i.e., modules, networking, kernel): Please do not put anything here. This is how Kernel.org uses an undocumented keyword system, which has nothing to do with Ubuntu Tags system.

[4.] Kernel version (from /proc/version):
While booted into the newest upstream mainline kernel only, please execute the following in a terminal and paste the results:
cat /proc/version

[5.] Output of Oops.. message (if applicable) with symbolic information resolved (see Documentation/oops-tracing.txt)
This is only relevant if you had a oops crash (ex. flashing Caps Lock light). While booted into the newest upstream mainline kernel only, if you have a kernel oops, one may consult http://www.kernel.org/doc/Documentation/oops-tracing.txt . If this is too daunting please ask for help in your bug report, and of the Ubuntu community.

[6.] A small shell script or example program which triggers the problem (if possible)
This is for advanced community members. If you feel comfortable enough to do this, please do it. It is not crtical if this is beyond your skill level.

[7.] Environment
Please execute the following in a terminal, and paste the results:
lsb_release -rd

[7.1.] Software (add the output of the ver_linux script here)
While booted into the newest upstream mainline kernel only, this is found in the directory:
/usr/src/linux-headers-<VERSION>/scripts

where <VERSION> is the version of the kernel you are using, found in the directory /usr/src. You may run the script by changing to the directory via a terminal, and paste the results:
sh ver_linux

[7.2.] Processor information (from /proc/cpuinfo):
While booted into the newest upstream mainline kernel only, execute the following in a terminal, and paste the results:
cat /proc/cpuinfo

[7.3.] Module information (from /proc/modules):
While booted into the newest upstream mainline kernel only, execute the following in a terminal, and paste the results:
cat /proc/modules

[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)
While booted into the newest upstream mainline kernel only, execute the following in a terminal, and paste the results:
cat /proc/ioports
cat /proc/iomem

[7.5.] PCI information ('lspci -vvv' as root)
While booted into the newest upstream mainline kernel only, execute the following via a terminal, and paste the results:
sudo lspci -vvv

[7.6.] SCSI information (from /proc/scsi/scsi)
While booted into the newest upstream mainline kernel only, execute the following via a terminal, and paste the results:
cat /proc/scsi/scsi

[7.7.] Other information that might be relevant to the problem (please look in /proc and include all information that you think to be relevant):
While booted into the newest upstream mainline kernel only, execute the following via a terminal, and paste the results:
ls /proc

[X.] Other notes, patches, fixes, workarounds:
Please provide a link to your Launchpad bug report.

As well, it is quite helpful that any further debugging information from http://www.kernel.org/doc/Documentation/ using the newest upstream mainline kernel available is provided.

In addition, please be 100% certain you have included all relevant comment information not included in the Bug Description and a kernel developer should review. They are some of the busiest people in the Linux space. They do not need to dive through downstream bug reports to find something that should have been provided when the bug was first filed. If you are unsure about anything asked for, or intend on not following this format, please do not file a report. Instead, continue to ask questions in Launchpad, and of the Ubuntu community, until all issues are cleared up.

Second step: E-mail the maintainer mailinglist with Kernel.org format

  • Please note, when E-Mailing a maintainer mailing list, do so in plain text, not html. Otherwise, it will be flagged as SPAM and bounced.
  • Do not post any of the Kernel.org information as an attachment when e-mailing a maintainer mailing list. Instead, provide the information word for word in the body of your e-mail.

Reporting USB bugs upstream

Reporting Wi-Fi (802.11) USB dongle bugs upstream

  • If you have a wireless USB dongle bug, using the Kernel.org format, please:
    • Send an E-Mail to the linux-wireless@vger.kernel.org mailinglist.

    • If this is a regression, CC the submitter of the regression commit. Also, CC those who have made recent (6 months to a year) changes to the module you are reporting about.

Reporting non-wireless USB dongle bugs upstream

  • First, please note that for non-wireless USB dongle bugs, the USB maintainer(s) do not want anyone to create a upstream bug report. For more on this, please see this upstream report comment. If you do open an upstream report anyways, it will cause negative unintended consequences. Instead, using the Kernel.org format, please:

Reporting non-USB bugs upstream

  • If the bug is not in USB, the first step in reporting a bug upstream is to find the maintainer of the driver for the bug from the MAINTAINERS list, or if a kernel graphics issue http://www.x.org/wiki/XorgMailingLists/. Then, using the Kernel.org format, please:

    • Send an E-Mail to the mailing list indicated by the 'L:' (use the maintainer E-Mail ONLY if no list is specified).
    • If this is a regression, CC the submitter of the regression commit, and the maintainer. Also, CC those who have made recent (6 months to a year) changes to the module you are reporting about.

Negative unintended consequences

  • If you contact upstream, please follow the kernel.org format created by kernel.org developers. Failure to follow these directions exactly as shown may have the following negative unintended consequences:
    • May likely result in your bug requiring upstream developers to ask unncessary follow up questions that should have been provided in the first place.
    • May likely result in your bug being promptly ignored by the kernel maintainer, submaintainer, or community developer(s) responsible to fix your kernel bug.
    • May create friction between you, the Ubuntu Community, and the kernel.org developers, which is not very Ubuntu.

    • May cause the report to be closed as REJECTED INSUFFICIENT_DATA, or REJECTED INVALID due to missing information or not following maintainer contact preferences.


CategoryKernel CategoryBugSquad

KernelTeam/KernelTeamBugPolicies (last edited 2014-01-28 14:59:30 by penalvch)