X/TestingInfrastructure

TestingInfrastructure

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

Launchpad Entry: x-testing-infrastructure
Packages affected: xorg

Summary

This specifies a basic framework to use for doing manual, semi-automated, and automated testing of X.org and related bits. Several side benefits include: a detailed debugging procedure handbook that will help Canonical support; tests that can be contributed upstream; and automated builds/packaging of newly released video and input drivers that users needing driver updates can easily access.

Release Note

[None: This feature is not distributed via the Ubuntu distro]

Rationale

Xorg makes many changes upstream which improve the code but cause regressions under some circumstances - such as with hardware that upstream does not commonly use, or when using non-default config settings. Unfortunately due to lack of a testing infrastructure upstream, these bugs crop up downstream at the point where we deploy the code to end users. We'd like to have mechanisms to quickly identify such flaws prior to deployment so we can report them upstream and avoid deploying code we know will cause breakage - or to at least have better data about what stability tradeoffs we're making when deploying a new piece of code.

Use Cases

Fabio is helping a customer troubleshoot a problem. He goes through the X testing documentation at wiki.ubuntu.com/X/Debugging and finds a procedure that addresses the issue. By running through it with the customer, they collect sufficient information that it becomes straightforward to fix.
Keith posts a new feature for X.org. It is automatically pulled down by the Ubuntu X testing infrastructure and packages are built and uploaded for Ubuntu users to easily install. Several Ubuntu users interested in the feature test out these packages and report their findings. By the time Keith's new feature is officially released, it's been so thoroughly tested by Ubuntu users that it works perfectly.
Bryce makes an error while fixing a bug in dexconf, but doesn't detect it before uploading the xorg package to his PPA. Thankfully, the Ubuntu testing system automatically downloads new X PPA packages and runs some basic pre-defined tests, and in doing so it notes that this new dexconf creates xorg.conf files that fail to allow X to load. Bryce quickly fixes the issue and is glad no one saw his silly typo.

Assumptions

Upstream's code does not churn in manners that require a large amount of hands-on packaging work

Design

Often when developers think of "improving testing", the first thing they think of is automated test harnesses, regression/compliance test suites, and synthetic performance workloads. Certainly those are valuable tools! Unfortunately they require a significant development effort, and can often result in less bang for the buck (in terms of fixed bugs), than if the time was spent just fixing bugs directly.

Instead, while leaving that as a longer term goal, this blueprint focuses on hybrid solutions that make current manual approaches more effective and more scalable.

The current process is weighted heavily on users who locate issues in deployed code during use. We can improve things on this end by giving these users better troubleshooting and bug reporting documentation. In the best cases, this should help them find a solution and send us a patch; in other situations this should at least enable them to report the issue with more specificity so the bug can be more quickly upstreamed (or fixed directly by a developer).

When a bug is forwarded upstream, developers often request that the user recreate the issue in the latest version of their code in git. For many users, this is beyond their technical ability. We can solve this at the distro level by providing daily builds of upstream code for them to test. We needn't build every upstream package, though, since only the drivers, mesa and xserver change with sufficient frequency to make these worthwhile.

One common class of bugs are regressions. If the user has found that one version has the issue and another does not, upstream will frequently recommend the user do a git-bisect to find the patch that introduced or solved the issue. (Indeed, if the problem does not occur in their git tree, they'll consider the bug fixed and leave it to the distro to isolate and backport the patch.) Unfortunately, git-bisecting is both beyond typical Ubuntu user's technical ability and time consuming. What we can do is pre-build a sequence of versions for the given codebase and provide .debs that can be tested, bypassing the need for each user to build the deb themselves. By eliminating the need for users to build packages, this opens testing to a much larger range of users, and leverages their time more efficiently.

Another common class of bugs are crashes and lockups. We already have a documented procedure for obtaining X backtraces, which we typically refer all crash reporters to. The process is relatively "paint by numbers" but involves a number of technical steps that may put off non-technical users. So, one way to improve reporting of crash bugs is to simplify the process - perhaps by simply improving the documentation, or by providing tools to automate part of the task, or perhaps even by improving the backtraces that the X server generates to begin with. In the case of lockups/hangs, where the server hasn't crashed but is not responding, we do not have a good procedure for debugging these cases; this needs researched and documented.

Implementation

The testing infrastructure can be divided out into a number of different components:

[DONE] X Debugger's Handbook - Documents the typical (manual) procedures for testing and debugging issues. This documentation will serve multiple purposes: a) education for new X bug triagers, b) help docs for advanced users with X issues, c) reference guide for Canonical support staff, and d) source material for future test writers.
- This handbook was written for Hardy. It is now in regular maintenance; most work will focus on expanding it to cover more use cases.
[DONE] git-commit-log - Generates a list of commit IDs for a given git repository appropriate for bisection searches, and in a fashion that git-pkg (below) can read.
[DONE] git-pkg - Given a git commit ID and an upstream git repo, pulls this version of the code and generates the packaging *.dsc
- [DONE] For Hardy, a proof-of-concept git-pkg was written specific to the -intel driver.
- For Intrepid, extend/abstract the script to be usable for all major video and input drivers.
- Later, extend/abstract script to work with mesa and the xserver
[DONE] auto-xorg-git - Automate the process of building git versions of upstream packages, including Debian and Ubuntu patches.
- [DONE] Main development of tool completed by Tormod and Bryce
- For Intrepid, investigate code sharing with git-pkg, and make it work more automatically
X Historical Package Pages - Script to generate HTML pages for each driver showing releases and git versions with packages for download
- [DONE] For Hardy static mockup was made: http://people.ubuntu.com/~bryce/bisect/
- For Intrepid, need to write a script to generate the page
- Need to extend to cover additional drivers, provide amd64 packages, and provide packages for debian/ubuntu releases
X Smoke Tests - A handful of simple test scripts to be used either manually or automatically, that test a few basic aspects of X (did it start? Is it in the expected resolution? Does a video play?)
[DONE] Backtracing - Improve user-collected backtraces
- [DONE] Improve backtrace documentation
- [DONE] Research how to debug Xorg lockups
- [DONE] Investigate if X server can be configured to provide more detailed backtraces. Update: Can't get more detailed backtraces via xserver, but can via apport now.