Triaging

Introduction

First, thanks for helping with X.org bug triage! X is a key piece of the Ubuntu architecture that *everyone* uses, and the bugs people experience tend to be pretty severe, so each and every bug you help move towards being solvable directly helps make Ubuntu better and easier for everyone.

The primary issue you'll deal with as a bug triager is that nearly all X bugs are reported without the information needed to classify and analyze the bug. This is where your work will make a big difference.

You don't need to be an expert with X to do this triage, as we've documented a lot of the common situations; so just regular Linux know-how will do.

You're welcome to dive right into the bugs at this point and triage away. You may already have a favorite X package to work on; if not, here is a list of all packages that the Ubuntu-X team maintains:

Triaging activities should include:

  1. Weed out non-X bugs (i.e. reassign kernel/gnome/plymouth/etc. bugs to appropriate packages).
  2. Move stuff out of the xorg package to the right X package.

  3. Set to Invalid if report doesn't define what the problem is, describes a non-issue or configuration problem, is just user confusion, or is otherwise unworkable or unhelpful to us.
  4. Verify reporter gave the right logs and steps to reproduce; else ask and set to Incomplete.
  5. Once all the needed info is there, set to Triaged and set its Priority.
  6. If the issue looks like something worthwhile for us to work on (e.g. affects a lot of users, or has an obvious fix), assign the bug to the canonical-x team for further handling.

The Perfect Bug Report

A perfect bug report begins with a perfect title. This should be constructed to indicate the chipset, features needed, short description of the problem, and condition(s) under which it appears.

  [i321] (Needs Acme) X freezes on typing Any key when TLA turned on

A good bug description would include several sections:

  [Problem]
  1-2 sentence description of problem

  [Discussion]
  Longer analysis or description of the problem (often just the original
  report with irrelevant bits deleted)

  [Workarounds]
  List of methods to work around the problem

  [System Environment]
  lspci, package versions, and other useful info on the hardware and
  software tested as affected by the problem.

  [Original Report]
  The original report text (if there's more discussion than what is
  above).

The bug report itself should include data files appropriate for the bug symptom. As a bare minimum it should include the Xorg.0.log. lspci -vvnn is also generally needed for video driver and xserver bugs. See X/Reporting for a more comprehensive listing of what is needed.

Ideally, the bug should also be tested against the newest version of the package that you can get your hands on, because if there is a newer released version, upstream generally will not respond to bug reports about an earlier release. For example, often newer versions will be posted to one of these semi-official Ubuntu-X PPAs:

Automatic Bug Triage

For Ubuntu-X, we have scripts to automatically do triage of several kinds of bugs for most X packages, so you do not need to spend time on these activities:

  • Processing new bugs - mostly automatic
  • Adding lspci info to descriptions - automated if lspci*.txt provided
  • Expiring old bugs - automatic

Package Classification

Most incoming X.org bugs get assigned to the xorg package since people don't know where to really file them. However, 99.9% of X bugs aren't actually bugs in this package, so we have to periodically go through and reassign them to the correct package. This is pretty easy, just time consuming.

Rules of thumb:

  • If bug mentions a particular video or input driver, assign to that driver. If they don't mention the driver, look in the Xorg.0.log for lines like "INTEL" (xserver-xorg-video-intel), "RADEON" (xserver-xorg-video-ati), "FGLRX" (fglrx-installer), "NVIDIA" (nvidia-graphics-drivers-180), etc. Most bugs belong here.

  • Crash/hang/freeze/lockup bugs should go to xorg-server if not associated with a specific driver.

  • High X CPU usage: Not generally an X issue, ironically enough. Refer user to https://wiki.ubuntu.com/X/Troubleshooting/HighCPU and set package blank, not xorg.

  • Keyboard layout bugs go to xkeyboard-config

  • Issues with hotkeys (e.g., power, media) usually aren't X bugs (many people mistakenly file them against X); they should usually go to hal-info instead. Refer user to http://wiki.ubuntu.com/Hotkeys/Troubleshooting.

  • Tablet bugs go to wacom-tools

  • Mouse and other keyboard bugs go to xserver-xorg-input-evdev

  • Font size/dpi issues should go to xserver. Reference - https://wiki.ubuntu.com/X/Troubleshooting/HugeFonts

  • If in doubt or for everything else, file to xorg and someone else will figure out where it goes.

Incomplete-With-Response Bug Triage

Do a query against 'incomplete-with-response' bugs.

  • Has the original reporter provided the requested information (usually Xorg.0.log and lspci -vvnn)? If so, set to Confirmed.

  • If they provided only partial info, or someone other than the original reporter commented on the bug, re-ask the original reporter to respond, and reset the bug, by setting status to New and back to Incomplete.

Title Fixing

Most people select poor titles for their bug reports. These can be improved pretty rapidly. The format to shoot for is:

   Title:  [CHIP] (Needs...) Problem with/when/on ... [TECH]

CHIP comes from the VGA line of the lspci info. You need this only for bugs that are filed against video drivers. Look for something like this:

01:00.0 VGA compatible controller [0300]: ATI Technologies Inc RV535 [Radeon X1650 Series] [1002:71c7] (rev 9e)

Here, CHIP is RV535. X1650 would also be acceptable. Best would be CHIP = "[RV535 X1650] ..."

For intel cards, CHIP will generally be something like "[i945]" or "[gm45]".

(Needs...) indicates that the bug needs a specific kind of fix, or some new feature we don't yet use by default. Usually you don't need this except in specific situations. For example, if the bug can be fixed with a Pipe-A quirk from X/Quirks, it should be "(Needs pipe-a quirk)", or if the issue is seen to go away when using some tech we don't ship as defaulted on, like UXA, you'd say "(Needs UXA)".

These statements are very helpful to include in the title because it helps the developers in making the decision about when to turn on a new tech, and flags bugs that can be closed when that new tech becomes available.

Problem with/when/on is the basic problem description. Often this needs rewording. Many people select titles that describe a generic symptom, like "corrupted graphics", or "black screen after install", which could occur for a number of different reasons. Improve them by looking for conditionals; often you can take them straight from the bug description. For example, "Corrupted graphics with dual head when using Xinerama on a Sony Vaio VGA-A270P".

Watch out for words like "randomly", "unpredictably", etc. - these usually indicate the reporter hasn't done enough to characterize the bug.

The reason revising the problem description is important, is because with generic titled bugs, other bug reporters will incorrectly assume they have the same bug and will "me-too" onto that bug rather than reporting their own issue; unfortunately this will result in them not getting their issue examined, and adds clutter to the original bug, making it harder and more time consuming to solve.

[TECH] indicates if the user found the issue with some non-default xorg.conf option or experimental technology. For example, if they are testing with Option "AccelMode" "UXA" in their xorg.conf, it would be [UXA]. If they are finding the bug when setting Option "DRI" "off", it would be [no DRI].

Don't include stuff like the driver name, e.g. "radeon driver fails..." (already implied by the package assignment), or the distro version name, e.g. "[jaunty]" (any bug open is assumed valid for the current development version; if not, "Nominate for release" should be used to target it properly.)

Confirmed Bug Triage

Once an X bug has made it to Confirmed status, it's generally got enough info and clarification to begin some basic troubleshooting.

  • Common issues: Refer to http://wiki.ubuntu.com/X/Troubleshooting/ for several guides on troubleshooting classes of X bugs that come up a lot. Often this will suggest further information needed, so request it and set the bug back to Incomplete.

  • Quirkable issues: Refer to http://wiki.ubuntu.com/X/Quirks/ for fixes to specific kinds of issues that come up a lot. These are easy fixes to get in.

  • Recent Regressions: If they noticed the behavior within the last few days after doing an system update, have them check their /var/log/dpkg.log to see what packages changed. They may be able to downgrade individual packages that are still present in /var/cache/apt/archives/, to determine which version change introduced the regression. This can be *extremely* helpful.

  • Dupes: Be careful setting dupes with X bugs, because different bugs often show identical symptoms. If in doubt, put in a comment asking "Is this a dupe of ...?" and leave it for someone else to decide.

  • Handle me-too'ers: Often people will comment on a bug that they too have the same problem. Usually we ask that they file a new bug report, unless they show strong proof that they really do have the same bug. This is because upstream has a "one-issue-one-person" rule, so if there's any chance the me-too'er has a unique bug it needs reported separately in order to get properly examined. If it really is a dupe bug, marking it a dupe is cheap and easy in launchpad. # dupes is also a good way to flag important bugs, whereas bugs with endless 'me-too' comments just are a drag to slog through.

If a bug is complete and ready to be forwarded upstream or solved in Ubuntu, or if there is a patch available, mark it Triaged (if you can).

Description Tidying

Many bug reports either a) are too terse, or b) ramble on too much. When we forward them upstream, the clearer the bug report's description is, the more likely it will be to get worked on.

A good bug description should have:

  • A 1-2 sentence [Problem] statement

  • Steps to reproduce
  • lspci info - the System Bus, and the Video Card(s)
  • Any unusual error messages or warnings in the Xorg.0.log or other files
  • Discussion about particulars of the bug (usually just reuse the text of the [Original Report])

Marking Bugs Ready to be Upstreamed

Often bugs are best solved upstream, so many of our bugs should be filed upstream with them. However, it's important to ensure we send them only quality reports that have a high chance of being solved. It's our job to filter out reports that lack information or otherwise are inappropriate for upstream. This work can be divided into two steps: Marking bugs for upstreaming, and filing the upstream reports.

A bug is ready for upstreaming if it meets all the following criteria:

  • The original reporter (or a secondary reporter who has proven to have *exactly* the same issue) is active on the Launchpad bug and can follow up on requests.
  • The issue has been verified using upstream's git-head of xserver and/or the relevant driver, or at least against the newest version of the code in the latest Alpha/Beta of the Ubuntu development release.
  • The issue is not Ubuntu-specific, and not likely to be a kernel issue.
  • The bug has complete set of log files, backtraces with debug symbols (if its a crash), config files, screenshots, etc. as appropriate.
  • If the bug is a regression, the last known good version, and the first known bad version.
  • The bug has a solid set of steps to reproduce it on demand (if it occurs randomly or intermittently, upstream may not accept it).

To mark a bug as ready for upstream, click "Also affects project", then leave the URL empty and click "Add to bug report", and "Confirm". This will leave a blank upstream task to search on later.

Reporting X issues upstream

To get issues addressed effectively, its important to provide complete information and clean, high-quality reports them. Indeed, much of the above bug work is geared with the objective of gathering sufficient information that upstream will be able to deal effectively with the problem.

Do a query on "Bugs that need reported upstream" (or bugs marked Triaged + not known upstream). X bugs are upstreamed to the bug tracker at http://bugzilla.freedesktop.org. Some guidelines to follow:

  • Focus each bug report on a single issue. Even if the Launchpad bug has multiple "me too" comments, only focus the upstream bug on one of these.

  • Attach complete logs, config files, screenshots, photos, and all other collected evidence.

  • Indicate that you are forwarding a Ubuntu bug, with a link to the bug in LP for cross-referencing.

  • Always tell the original reporter that their issue is forwarded upstream, and ask them to subscribe to it.

Here's a boilerplate for the bug report:

I'm forwarding this bug from a Ubuntu reporter.
[Launchpad Bug URL]

[Problem]
...1-2 sentences...

[lspci]
...Bus ID (lspci -vvnn | head -n2)
...VGA IDs (lspci -vvnn | grep -A1 "VGA compatible")

[Original Report]
...

Here's a boilerplate to tell the original reporter:

Hi, I've forwarded your bug upstream to [freedesktop url] - please subscribe to this bug in case upstream needs further information or wishes you to test something.  Thanks ahead of time!

Upstreaming AMD fglrx issues

fglrx Required Information for upstreaming

  • Xorg.0.log
  • lspci -vvnn
  • Steps to reproduce
  • (For crashes) Backtrace
  • (For screen corruption) Photo/screenshot

It is important that the issue be reproducible when upstreaming, because AMD needs to be able to reproduce the issue in-house in order to be able to accept it.

fglrx Bug Upstreaming Procedure

To mark a bug as needing to be upstreamed to AMD:

  • Click on the "Also affects project" link
  • (If necessary) Select or type in "fglrx". Usually this step is automatic.
  • Tick the third bullet, "I just want to register that it is upstream right now" and click continue
  • (Optionally) Set the priority of the 'fglrx' bug task as appropriate.

This marks the bug as one to be sent upstream.

fglrx Upstream Statuses

  • New - Needs communicated to AMD.

  • Incomplete - AMD needs more information to continue.

  • Confirmed - It has been communicated to AMD, but we await their response.

  • Triaged - The issue is accepted into AMD's internal feature or bug tracker. Usually we should have a feature or bug ID number at this point. This number is recorded in the bug title.

  • Won't Fix - The bug is recognized as legitimate but will not be acted on upstream or is not of interest upstream.

  • Invalid - The bug report is not valid for some reason, such as: Cannot Reproduce, Expired, Obsolete, Not an Upstream Bug, Other.

  • Fix Committed - The issue is fixed, but not yet available in a public release of the driver[*].

  • Fix Released - The issue is fixed and available in a released version of the driver.

* - Note that due to NDA requirements, we may not be able to provide status information on some bugs until the fix is publicly available.

Upstream NVIDIA issues

See http://us.download.nvidia.com/XFree86/Linux-x86/1.0-9755/README/chapter-06.html


Also see: Bugs/HowToTriage CategoryXTeam

X/Triaging (last edited 2016-01-10 19:32:51 by penalvch)