Bisecting

If you were asked to bisect the X.org stack to identify the regression commit, were referred to this article, and you are willing to do so, thank you for your efforts! You are taking the best route to get your bug resolved as soon as possible.

What is a bisect?

A forward bisection, or just traditionally called a bisect, is the fastest process of finding the midpoint between a known good software version release, and a known bad one released afterwards. One continues finding the successive midpoint until one identifies the last good software version release, followed consecutively by the first bad one. Ideally, if one could determine a set of commits that are known good and known bad within a larger range, one could reduce the number of bisect iterations required. However, a bisect would be handy when this wouldn't be possible.

Performing a bisect is faster than testing every version in between the initial known good version, and the known bad one. For example, if your known good release was 1.0, and bad was 1.10, the worst case scenario would be testing 9 releases (1.1, 1.2, 1.3,..., 1.9) until 1.9 was found to be the last known good version. However, bisecting the worst case scenario, one would test only 4 releases (1.5, 1.7, 1.8, and 1.9).

Bisection is a good technique for finding the exact change that a) introduced a bug, or b) solved a bug. Thus it is useful if you wish to solve a regression or backport a patch to fix an issue.

How do I bisect an X.org bug?

For example, let us say you started with a fully updated Precise install. Then, instead of upgrading you just did a clean install of Trusty, and found what you think may be an X.org bug, that was not present prior to Precise. However, if you are unsure of what release this regression occurred with, the next step would be bisecting Ubuntu releases.

Bisecting Ubuntu releases

Hence, one typically wants to narrow down the first Ubuntu release after Precise this problem began in. So, we have the following releases:

Ubuntu 14.04 Trusty Tahr
Ubuntu 13.10 Saucy Salamander
Ubuntu 13.04 Raring Ringtail
Ubuntu 12.10 Quantal Quetzal
Ubuntu 12.04 Precise Pangolin

The midpoint release between Precise and Trusty is Ubuntu 13.04 Raring Ringtail. One may download releases from http://releases.ubuntu.com/. If the bug is not reproducible in Raring, then one would want to test Ubuntu 13.10 Saucy Salamander. If this is reproducible in Saucy, then one knows that the regression happened going from Raring to Saucy. The next step would be commit bisecting X.org between two versions.

Commit bisecting X.org between two versions via git

For Mesa-specific problems, go here: X/BisectingMesa

It's recommended that you do this only on a development box (or a virtual machine) you can reimage when you're done, since you'll be installing stuff outside Ubuntu's normal package management system.

Continuing the above example, lets assume the issue is a backtrace crash of the xorg-server package. As per the package page, one knows the version of xorg-xserver released in Raring was 2:1.13.3-0ubuntu6, while Saucy was 2:1.14.3-3ubuntu2.

one would install git, and clone the upstream xserver repository via:

sudo apt-get -y install git-core gitk
git clone git://anongit.freedesktop.org/xorg/xserver
cd xserver
git bisect start
git bisect good xorg-server-1.13.0
git bisect bad xorg-server-1.14.99.904 

This will produce the following in the terminal:

Bisecting: 434 revisions left to test after this (roughly 9 steps)
[1e6cf8ec20d07b73a11116564aba71b4e4291dcd] Merge remote-tracking branch 'jturney/unused-but-set-variable-warning-fix' 

Next, one would build this test commit following building X from source. If tested to reproduce the problem, you would then mark it via git:

git bisect bad 

Then you would continue bisecting until the offending regression commit is identified.

If you are unable to compile a particular revision (say it's picked one that introduced a compile breakage), type git bisect good, even though it is not known if the problem was reproducible or not, and then continue bisecting. The manpage for git-bisect has some more in-depth details.

One may utilize this bisection method to bisect other parts of the X.Org stack (intel, radeon, nouveau, etc.).

Notes

As a good rule of thumb when choosing your starting bisect points, upstream often branches the release candidates, often you'll want to back up some extra time - a few weeks in the case of a driver, or a few months in the case of the X server.

The X server (and to a lesser extent the -intel driver) are branched prior to release, which can make it tricky to find bracket points. If you're having trouble reproducing, try checking out the release branch and testing on that.

Ubuntu often ships X.org packages with some patches beyond what upstream ships. If you cannot reproduce in upstream git, try testing the patches in the Ubuntu package's debian/patches/ directory. A bisect-like approach would be to comment out half the patches in debian/patches/series, then build and test, and either disable or re-enable as appropriate.

Upstream periodically changes the dependencies for X components. If you're having trouble reproducing, you may want to explore also backing down some dependency versions such as mesa, libx11, etc.

Other Resources

X/Bisecting (last edited 2014-01-25 19:40:33 by penalvch)