FriendlyGitBisection
Background
Sometimes a bug in a current version of Ubuntu is found to not exist in a newer or older version of that package. Something changed that resulted in the breakage (or fix).
git bisection (see kernel bisection and X.org bisection) is a highly effective technique for narrowing down where and when the break began (or ended).
However, this process is technically challenging for most users. In addition to understanding the problem, it requires knowing how to use git, how to build the kernel or X.org pieces by hand, and how to install and run them. Very few users have this level of technical experience, and those that do often don't have the time to follow through with it to find the actual patch that brings the fix. (Once they have a kernel that works for them, why would they spend all that extra time building broken kernels just to figure out a fix for others?)
If users were able to do bisection searches more easily and include the results in their bug reports, it would make a lot of user-affecting bugs much easier and faster for developers to fix. Ubuntu's quality is significantly improved.
Concept
The idea is to strive to streamline the process by automating the builds and providing a friendly interface for users to browse builds and track the progress of their searches.
Tools are provided so that package maintainers can customize what packages are provided to the users.
The user can browse all the builds in a specialized PPA, or be guided by launchpad as to which package to try next. They can also associate their search with a bug report, so others can follow along.
When the user has successfully completed a search, the system makes it easy to copy the patch over to the bug, mark it as a fix, and flag the known-good package snapshot(s) as a temporary workaround for other users. It also makes it easy to extract the patch and build a test package with that patch alone applied to the current Ubuntu package to confirm whether that patch alone will fix it.
Use Cases
Robert has upgraded from natty to oneiric but now his system fails during log in. He finds that with the natty version of xserver-xorg-video-intel the system works. Using FriendlyGitBisection he is able to narrow the fault down to the 'break-roberts-login' patch. He files a bug report, the patch is swiftly reverted, and Robert is back in business.
- Francis follows the development version of Ubuntu religiously and updates daily. One day his wireless breaks after upgrading from kernel -88 to -89. Rebooting into -88, he downloads a dozen snapshot builds of kernels between those two versions and sets to work testing them. 3 of the 12 show the failure. He shares this info with a kernel developer, who immediately spots the problem, and queues the fix for -90.
Victor receives a panicked call from a customer who just received 100,000 new desktops, that don't boot with the Ubuntu LTS. Victor thinks he's going to need to hop on a plane, but decides to try walking the customer through use of FriendlyGitBisect. The customer is receptive and over the course of the afternoon on the phone, booting dozens of kernels, the two are able to narrow it down to a change that introduces several PCI IDs, one of which corresponds to a newly released device included in the customer's machines. Victor's no kernel engineer but after creating, testing, and verifying a kernel build with just that patch alone he's able to file an SRU and get the fix into -proposed that same day. The customer is relieved this was resolved with no impact to their rollout schedule.
Zöe maintains a derivitive of Ubuntu which has customizations for improved media playing. Unfortunately, after syncing the latest packages from Ubuntu things aren't working. She suspects one of the custom options they set may have suffered a regression since the problem doesn't exist in Ubuntu. She sets FriendlyGitBisect to provide builds with that option enabled and another set with it disabled for testers to experiment with.
Design
For a given source package, snapshots are taken at various points in the revision tree between two points as specified by the package maintainer. They could choose to do builds of every revision, every Nth revision, once a day, or a hardcoded list of revisions.
Not all revisions are expected to build; those that don't are marked as such for the maintainer's review. It is also possible some revisions will build but will be broken to a degree that they can't be used for testing; the user is able to provide feedback to this effect, so as to help warn other users of the potential problem.
A package maintainer can set up multiple start-end sequences for a given package. For a given sequence, the package maintainer provides a debian packaging directory to apply on each snapshot. The system will automatically add a changelog entry but will not make any other changes to the debian directory for that sequence. Different sequences can have different debian packaging; this permits the package maintainer to adjust for changes in dependencies, or to establish several sequences with different build options enabled or other changes applied.
A user can start a search session by specifying known-good and known-bad points. They can also (optionally) associate a search with a given bug report.
When a user selects a package and installs it, the system will take note. On a subsequent visit, the user will be prompted to indicate whether that version displayed the bug, behaved properly, or was broken/unusable for some other reason. Based on this, the system suggests the next package to test. If the user has associated their search with a bug, their status and progress will be made visible when viewing the bug report.
The system also provides a list of the pre-built packages, to permit the user to manually install specific versions or to download several packages in one go (e.g., so can do offline testing).
When a user has completed the bisection search to the degree the system permits, they are asked if they wish to link their findings to the bug report.
Prototype
I (Bryce) did a simple prototype of this concept years ago for the -intel DDX video driver. I pre-built .debs of every 4th git revision between what we shipped in two versions of Ubuntu. I identified bug reports that indicated regressions from the earlier to the later version, and had the reporters use this tool to narrow down the fix.
Results
I have to admit in none of those cases did this result in us finding a fix. In most cases it turned out the bug was actually due to changes elsewhere (libdrm, kernel, mesa, etc.) If I'd set the tool up to cover those packages as well, we might have been able to continue the hunt.
But, despite that, users found this to be a very pleasant experience. Most of them successfully made it through the full process, and did got results back to me quite quickly. It definitely brought a better experience.
The failings of this tool I felt were not problems with the concept, but a) technical challenges with non-linear git histories, particularly around transition points, b) running and managing all the builds myself on my own hardware was hard to stay on top of, and c) the number of commits for the -intel driver is small compared with mesa or the kernel, but to expand to cover the full stack would be much more extensive.