PreventingHardwareSupportRegressions

Summary

Implement a testing and debugging strategy that will allow faster response to hardware regressions. These regressions cannot be prevented, but with the proper infrastructure, can be caught faster, and thus fixed in a smaller timeframe.

It is also hoped that the hwdb can be used to preemptively call for testing of specific drivers against users who are known to have such hardware, and are willing to test these upgrades.

Rationale

We frequently have regressions in the kernel regarding hardware support. Drivers will stop working, or crash. Hardware will suddenly not be recognized, or fail to work as well as it did before. This all happens during the development phase. Assuming this, our goal is to make sure that all regressions are fixed prior to the release, so that someone upgrading from one release to the next will continue to have working hardware.

At the very least, release upgrades should work as well as the last. In some cases we want them to work better, such as supporting hardware that was not supported before.

The hope is that a proper testing and debugging strategy, in addition to educating those users that follow the development cycle, will enable us to react more quickly and with better results to the regressions.

Scope

This spec addresses kernel testing. It should not be confused with generic hardware testing. Each regression is important and testing on all possible setups is vital for kernel stability. No single subsystem is more important than another. Any regression is important. Again, this is not a spec for general hardware testing. It does not cover cases where a driver was always broken, or the hardware was never supported.

Use cases

  • Mary upgrades from breezy to dapper, and because of our extensive testing, all of her hardware continues to work at least as well as it did in breezy.
  • Kernel maintainer wants to upgrade the rt2500 driver, but wants a list of testers to contact prior to finalizing the change.

Implementation

  • Make community aware of the situation using appropriate media (wiki as a general reference point and mailing list for the initial announcement)
  • Provide detailed instructions to users for how to test for and report regressions using the daily builds via the wiki so that they can be easily updated:
    • Where and how to download daily builds. (see below)
    • Using the daily builds to find the exact version of the kernel where the problem started to exist.
    • How to download and use the "debugging" options. (see below)
    • Templates for filing bug reports.
  • Call for testing via mailing lists on kernel-team@. While this practise will be used in the beginning to underline specific changes in the kernel (impossible to define at spec writing time), we want testing to become part of a common developer day.

  • Make use of the hwdb in order to find testers for hardware-specific changes in the kernel. Subject would contain basic information for type of hw affected and pointers to wiki templates for reporting both success/failures. (HardwareDatabase)

  • Provide a daily build of the kernel from git (see GitRoadmap) in order to get a more fine-grained idea of the changesets that caused the regression. (Note: GitRoadmap has not been written down, but the general idea is that, except in rare cases, the git repository must always be buildable).

  • Provide a debugging daily build of the kernel from git that people can use to provide more detailed output to the kernel team via the mailing list or bug reports.
  • Implementation will be performed by the kernel team.

Code

  • We need daily build infrastructure

  • hwdb modifications have been already discussed and planned with Oliver and they were already part of other hwdb specs. (HardwareDatabase)

Outstanding issues

Most of the time we can't prevent breakage, but we can detect it earlier and try to fix it more quickly.

We don't want to ask users to install dapper too early, as we don't support it. The live CD would be a very good tool for testing but it requires the kernel in the archive (plus propagation to the CD) and it is an expensive operation for a simple round of testing. Also some people think the live CD is too big to download.

Comments

  • You could add an "I want to help testing hardware" checkbox + an e-mail field (or launchpad ID field) to the hwdb application.
  • Downloading the whole live-cd several times a week/month is impossible for many people because of ISP-imposed restriction. But maybe an easy to use live-cd-builder based on jigdo or something like that can help reducing the required bandwidth?

CategorySpec

PreventingHardwareSupportRegressions (last edited 2008-08-06 16:28:35 by localhost)