FriendlyGitBisection
Background
Some bugs are hard to analyze, but you know they exist in one version of a particular package, but not in another. Clearly, something changed that resulted in the bug appearing.
An effective way to analyze these bugs is 'git bisection'. Essentially, search through the revision history to find exactly where and when things started breaking. Both kernel bisection and X.org bisecting are documented and done quite often.
However, this process is technically challenging for most users. In addition to understanding the problem, it requires knowing how to use git, how to build the kernel or X.org pieces by hand, and how to install and run them. Very few users have this level of technical experience, and those that do often don't have the time to follow through with it to find the actual patch that brings the fix. (Once they have a kernel that works for them, why would they spend all that extra time building broken kernels just to figure out a fix for others?)
Concept
The idea is to strive to streamline the process. We automate as much as possible. We give a friendlier UI than a git command line; either web or GUI or both. We pre-build packages to save on build time. We help the user keep track of what they test and maybe keep tabs on their progress centrally for our own reference.
Prototype Results
I (Bryce) did a simple prototype of this concept years ago for the -intel DDX video driver. I pre-built .debs of every 4th git revision between what we shipped in two versions of Ubuntu. I identified bug reports that indicated regressions from the earlier to the later version, and had the reporters use this tool to narrow down the fix.
I have to admit in none of those cases did this result in us finding a fix. In most cases it turned out the bug was actually due to changes elsewhere (libdrm, kernel, mesa, etc.) If I'd set the tool up to cover those packages as well, we might have been able to continue the hunt.
But, despite that, users found this to be a very pleasant experience. Most of them successfully made it through the full process, and did got results back to me quite quickly. It definitely brought a better experience.
The failings of this tool I felt were not problems with the concept, but a) technical challenges with non-linear git histories, particularly around transition points, b) running and managing all the builds myself on my own hardware was hard to stay on top of, and c) the number of commits for the -intel driver is small compared with mesa or the kernel, but to expand to cover the full stack would be much more extensive.