The goal of this spec is to produce a set of scripts which will allow the kernel team to mine data from the HWDB.

Release Note

A series of HWDB scripts have been uploaded to lp:~adeuring/+junk/hwdb-scripts. These scripts provide the ability to extract hardware information from a subset of the Ubuntu user base who have chosen to upload their hardware profiles. This data can help determine statistics such as the number of users of a specific device and bugs reported against that device. Those metrics can help prioritize work based on the number of affected users with that hardware.


Leading up to the Jaunty UDS the Ubuntu kernel team had been holding discussions with the Launchpad team to investigate building a hardware database that would allow end users to submit their hardware profiles to Launchpad. In theory, those profiles could then be linked to bug reports or have their data mined in general to determine how widespread a hardware specific issue may affect the overall Ubuntu user base. Having an API/scripts to extract this data would help the Ubuntu kernel team prioritize work to be done and resources to be allocated. The API/scripts could also help with the overall kernel bug triaging process. They could be used to quickly determine which bugs relate to a specific piece of hardware or driver. This could then facilitate advanced warnings to users for known issues, calls for testing, or grouping issues to a master bug.

An initial set of API's/scripts were written during the Jaunty development cycle but there remains areas of improvement.

User stories

  • We ran into a real world use case with bug 359392 for Jaunty. It would have been helpful if we could have queried all hwdb profiles to discover who had an i965 graphics controller [8086:2a02] and how widespread this issue would have been for the Ubuntu user base.
  • Another real world use case was bug 263555 - "2.6.27 e1000e driver places Intel ICH8 and ICH9 gigE chipsets at risk". We could have queried the hwdb for affected users and sent them an advanced warning email notifying them of the issue.
  • The kernel team would like to query all bugs tagged suspend/resume and return a list of bugs which use the rtl8187 driver. See the ubuntu kernel-team ml thread "Commonly found hardware in suspend/resume bugs.


All scripts will be written in python and use launchpadlib. In particular, refer to the hwdb api



Some scripts have already been written and are available at



    • Search for users owning the given device or having a device controlled by the given driver, and return Ubuntu bugs these people are subscribed to. You must specify a driver name, or bus, vendor ID and product ID.
    • Examples:
    • python -d iwl3945


    • Queries the entire hwdb and returns total number of devices owners, submitters, submissions etc.
    • Examples:
      $./ -nor -b PCI -v 0x8086 -p 0x2a02
      device owners: 45301 all submitters: 877776
      submissions with device: 55983 all submissions: 1179316
      total number of devices:  56537
      • Query only HWDB reports for Intrepid:
        • ./  -b PCI -v 0x8086 -p 0x2a02 -s intrepid

      • Only intrepid/amd64:
        • ./  -b PCI -v 0x8086 -p 0x2a02 -s intrepid -a amd64

      • You can also limit the counts to a driver and/or a (kernel) packagename, though this returns 0 for this device. Seems that most HWDB reports do not relate any driver with it. You'll get a bit more interesting driver/package-related numbers for the PCI device [0x10de:0x0429] (an NVidia graphics card), for example.

    • Provides ability to search for bugs related to a device or a device driver
    • Examples:
      • Search for bugs tagged "suspend" or "resume" and have a hw profile which references the rtl8187 driver
        ./ -D rtl8187 -t "suspend resume" -r -T Any
      • Search for bugs tagged "suspend" and "resume" against the 8.10 distroseries and use the rtl8187 driver
        ./ -D rtl8187 -t "suspend resume" -r -T All -S 8.10
      • Same as above but restrict against the linux source package
        ./ -D rtl8187 -t "suspend resume" -r -T All -p linux
      • Can also search against a specific device
        ./ -d PCI:0x8086:0x2a02 -p linux -r

Current Unresolved issues

  1. Doesn't work on Jaunty.

  2. Performance. It doesn't matter how good the documentation or APIs are if it takes too long to get the information out. Queries should be taking seconds, not hours. For Example: Running ` -b PCI -v 0x8086 -p 0x2a02` to see how many people may be affected by the i965 freeze issues (bug 359392) took 10+ hours to run! I actually don't know the exact total run time because I walked away after 10hrs and came back in the morning to see if it had finished.

    • We should indeed provide a number of API methods. Queries "canned on SQL level" should be quite fast compared with the existing scripts which need thousands of API calls to achieve a result.
    • Abel has a branch which should improve the performance of the scripts he's currently written. ETA for branch being pushed is Fri 12, June 2009

    • Branch has been pushed

  3. The main issue is the lack of linkage between bugs and hw-info. From what Abel has explained, the hardware information is submitted via checkbox while the bug reports are generated by apport (or filed through launchpad directly). The two utilities don't seem to know about each other. The linkage between hw profiles and bug reports needs to be bi-directional. Also, the way that existing scripts try to match up hw profiles to a bug doesn't take into account that a single user could have made multiple hw-info submissions related to different systems (desktop vs. laptop for example).

    • Right, the best option would be to automatically create a HW profile when apport reports a bug and to link the profile and the bug. That's something we should discuss during UDS, I think. This next-best option would be to allow bug reporters to link a bug with a hardware profile via a web UI. The problem here is that ca. 1/3 of all people who provided at least one HW profile actually submitted two or more profiles: It can be quite difficult to figure out which HW report belong to the machine affected by a bug. Martin had some interesting ideas how we can present the HW profiles on a web page, but I am not sure if/when we will have enough "developer capacity" to implement this. The third option is to provide a webservice API method which creates these links (would be also needed for option 1), and to write a script which creates these links. I think we should discuss options for better ways to link bug reports and hardware profiles at UDS in Barcelona. There are so many different people involved -- the QA team, checkbox developers, apport developers, LP developers -- that a meeting during UDS it is probably the best way to get something started.
    • Issue being tracked in Bug 379393 (Fix Released) - Currently launchpadlib has the ability to link hw profiles to bug reports. schwuk to update checkbox to store submission id (target karmic-alpha3 for completion of this). pitti to then update apport to use the submission id to link the profile and bug report.

    • Also see bug Bug 424382 (Fix Released in apport, Fix Released in Launchpad) - "FileBugViewBase should detect the header HWDB-Submission in data sent by Apport"

  4. The api documentation is pretty poor. For example, you have to just "know" that most if not all properties of "team" are also available for "person". The documentation doesn't indicate that in any way. Bug 363440 was filed a bug about this.

  5. Bug 338772 - Data from lshw should be submitted to Launchpad

    • 20090713 - Abel to start a discussion with Marc and Dave about this. Below are notes from Abel:

      1. I am not very familiar with the details how HAL will be replaced, but as I understand it, the successor is DeviceKit, which will provide similar data. So we could switch to DeviceKit. This would not solve the problem that server installations don't have HAL installed, or anything else DBus-related,

      2. use lshw. The problem here is that lshw does not list for example any USB device. This might not be a big issue from a kernel developer's viewpoint, but getting statistics about printer or scanner usage would be fine. And things like bug linking can also be useful for application developers. (And USB-related kernel drivers can also have bugs Wink ;)

      3. checkbox could traverse sysfs and use its own code to find the relevant data.
  6. Bug 399319 - Remove the HAL dependency from Launchpad HWDB and checkbox

    • 20090714 - Abel, "I think we should not use lshw as a replacement for HAL data, but the HWDB client can of course include lshw output and Launchpad can store and present it."
  7. 20090713 - Push scripts to a central hwdb project repository

    • Abel starting the process to get permission to start the project

Script Requests


Use Case



Linking profiles (more specifically devices/drivers) to bug reports

The majority of the kernel team's use case revolves around the ability to tie a profile (specifically a device/driver) to a bug report. It helps prioritize bugs and day to day triaging


We currently have the ability to link a hw profile to a bug (See bug 379393 and Bug 424682). Still working on linking the actual device/driver within a profile to a bug.

I don't believe it is currently possible to search based on something like system.hardware.vendor="Dell Inc." and system.hardware.product="Inspiron 1420". I think the closest that can be done is specifying something like vendor_id=0x8086.

I want to examine all bugs tagged "suspend" and determine which hardware vendor and model (eg. system.hardware.vendor="Dell Inc." and system.hardware.product="Inspiron 1420") each bug is being reported against. This will help to determine possible duplicate bugs and ensure they are duped to one master bug.


This is currently possible. We store these properties as the vendor/product IDs for the "bus" called "System". In the "real world", this can nevertheless be a problem with many IBM/Lenovo laptops. For these machines, system.hardware.product is something unreadable like "6457BAG", while the "ordinary" product name is stored in system.hardware.version.

request from Bryce to run device-statistics on the following devices. Basically it is all the Intel chipsets, which are those numbered [8086:*].

It would be interesting to see which chipsets are in heaviest use amongst Ubuntu users.


Restrict hwdb searches to just a bug *reporters* profile

I just want to search for bugs that have a reporter with the specified hw. I don't care about the subscribers of the bug since some people subscribe to bug just to monitor a bug but don't have the affected hw. Also, many times people have the same symptom of a bug but in reality they don't have the same hw so it's a different bug.


Investigate. This might be possible if we use and then use use bug.owner_link to check if bug.owner_link has the specified hw

Restrict hwdb searches to just the bug *subscribers*

I can take a known hw specific bug and find subscribers who need to open a new bug because they have different hardware. Otherwise they just end up spamming the bug with unrelated comments and making it more difficult for developers and triagers to follow the relevant issue.


Investigate. This should work like the following: 1) take a bug number 2) loop over all bug subscribers (or people saying they are affected by this bug) 3) show data about all hardware profiles from these people (This should include filters for this data, like "show only PCI devices" or "show only the device drivers")

Restrict hwdb searches to a specific package a bug was filed against.

A person may have submitted a hw profile but I only want to see bugs they reported against the "linux" kernel source package that the profile may be related to. I don't necessarily want to know about bugs they filed against say xorg for example.



Need to get authorization from people who submit hw profiles that it is okay for us to use the data they've provided and more importantly contact them regarding their hw.

I noticed that bug 359392 has the potential to affect a large number of users. A query of the hwdb validated this concern. It would be great if we could then be able to contact the affected users to issue calls for testing or give an advanced warning of the issue.


Not Complete

Intermixing search options, I think this could be resolved through a better api.

I'd like to search for all subscribers of a bug 359392 which have the i965 graphics controller [8086:2a02]. Or I'd like to search for all bugs tagged "suspend" whose bug reporter has a graphics card which uses the nvidiafb driver.


See and

Need to be able to expire profiles.

Bug reporter's hw has died.


Not Complete

I don't want to have to specify a bus type when searching for a vendor and device id

Use Case: It's just annoying specifying a bus type, it should be able to query across all buses


You might end up getting results for USB devices when you are looking for a PCI device, so I'm hesitating to remove this requirement.

Find the top 10 drivers with the most bugs assigned against them in the last 30 days.

This is a very interesting use case. But consider how slow the current scripts are: They either start with one given device (a driver would be the same, speed-wise), look up device owners and then and then look for related bugs; or they start with one given bug and then list devices from affected users. With the currently available webservice API, we would need to loop over a number of bugs, find affected users and finally look up drivers in hardware reports from these users. I think this use case really needs (but for sure deserves Wink ;) an "advanced" API method.



The Documentation Team has proposed/suggested that it would be nice to consider exposing the information from the hwdb in a more palatable interface than strictly an API.

The only alternative right now are the HardwareSupport section under the ubuntu wiki. And under the Hardware section in the community help wiki. This information has become out-of-date, poorly maintained, too large to manage, and often inaccurate. Instead, perhaps we could create a cron to mine the relevant data from the hwdb and program it to generate data that would be accessible through a web interface. The Documentation Team has suggested/envisions something similar to "". Once this becomes live, the Documentation Team would like to be made aware/informed so they can remove all hardware(specifically outdated and inaccurate) references from within the Wiki, and direct users to this proposed new database site instead. They have offered their assistance in helping to get this off the ground in any way they can as this could be a joint project. This may sound "too" ambitious, but perhaps this proposed/theoretical website, when searching for a particular piece of hardware, could also search launchpad for known bugs and the forums for related posts. This would help centralize all hardware related documentation while still keeping them semi-separate. For more information on this topic, check out:


Not Complete, unfortunately at this time the api is not in a state where this would be easily achievable. Additionally the resources available to develop such and interface is not available at this time. While it is recognized this would be nice to have, it's not possible at this time.

BoF agenda and discussion

Translated into table above.


specs/KernelKarmicHWdbWorkshop (last edited 2009-11-30 20:06:53 by c-24-21-43-9)