From The Art Of Community by O'Reilly (http://www.artofcommunityonline.org) by Jono Bacon

Ubuntu Bug Workflow

Technical collaboration has similar components and similar results. The mechanics of collaboration are so important because when you understand what conversations occur, you can optimize how people converse. Take bugs, for example. Every piece of software ever written has bugs. Software is written by people, people make mistakes, and bugs are human mistakes formed as bits and bytes. Most software projects use a special piece of software called a Bug Tracker to have a bug conversation. Someone files a bug. Someone else asks for more information on the bug. Those who are affected by the bug offer clues and information. Someone else may propose a fix. People try the fix and provide feedback. The essence of conversation is the same: a shared language (English), a shared topic (exploring and fixing the bug), and a communications channel (the Web).

When considering how to build your community’s infrastructure and tools, you need to identify what these mechanics of collaboration are. You need to think carefully about how people have conversations, both in natural language (such as discussions) and technical (such as with bugs). When you understand the driving forces behind how conversations occur, you can make some really interesting things happen. A while back my team spent some time working on optimizing the conversation around bugs. Before I explain the solution, let’s take the problem for a spin. In the open source world bugs are a complex topic, and they are particularly complicated for Linux distributions. Consider the problem. Erica is using Ubuntu and experiences a problem in the Text Editor: when she tries to spellcheck, the editor crashes.

As an experienced open source citizen, she is familiar with the mantra that when something goes wrong, you should report it. She clicks the Help menu in the text editor and clicks on “Report a problem...” to report the bug. At this point she now expects someone in Ubuntu to tend to the bug report and hopefully fix it. So far, so good. Now it gets a little more complicated. The Text Editor in Ubuntu is actually called GEdit. It is an open source project that is part of the GNOME (http://www.gnome.org/ ) desktop. The GNOME project is an independent project that produces software, and the open source parlance that describes this kind of project is an upstream project. Ubuntu developers take the upstream GNOME source code (including GEdit) and build packages to ship it in Ubuntu. When this code is built, some modifications are made to make GNOME (and as such, GEdit) integrate well in Ubuntu. This includes consistent themes, file dialog boxes, folder locations, and more. In a nutshell, Ubuntu takes the original upstream source code, adds some changes, builds it, and ships the final product as part of Ubuntu.

This raises our first question: who is responsible for the bug? Was the bug present in the original upstream code from GNOME? Was it in the changes made by Ubuntu developers? Does the bug exist outside GNOME and GEdit? Does the bug even exist at all? Fortunately, this is a fairly simple question to answer: some quick checking of the upstream code can usually identify the source of the bug. The second question is the one that is pertinent to our bug conversation. How do we deal with the bug report? Erica rightly reported the bug, but should she report it in the Ubuntu bug tracker (called Launchpad) or the GNOME bug tracker? In addition to this, many other people may have experienced the bug. Some will report it in their distribution’s bug tracker and some will report it in the upstream bug tracker. It is not entirely inconceivable that there is a bug report in each distribution’s bug tracker and one in the upstream bug tracker all pointing to the same bug. This is hugely wasteful.

Regardless of all this duplication of effort, if the bug exists upstream and the distribution ships it, the bug does still exist in the distribution and it does exist upstream. The challenge is to bring each of these different bug conversations together so everyone who experiences the bug can share their experience to fix it. The solution to this problem is to identify how the flow of conversation and collaboration happens between these different bug reports and to optimize it.

Fortunately, software can address these issues. Launchpad, the web service that Ubuntu uses as a bug tracker, already had the ability to link bugs, whether they’re in the same tracker or a different one. When Erica filed her bug, she or someone else could connect her bug to an existing bug in an upstream bug tracker. It works like this:

1. If the developer knows the bug is an upstream bug but does not know which bug it is in the upstream bug tracker, he can add an upstream task to the bug report. This upstream task indicates that the bug should be linked to an upstream bug when it is found.

2. If the developer knows the bug in the upstream bug tracker matches the Ubuntu bug, he can link the two bugs. This involves finding the Ubuntu bug and using the Link feature in Launchpad to enter the URL of the upstream bug.

When a bug is linked, any changes made to the upstream bug are synchronized in Launchpad and vice versa. This feature connects the two separate conversations.

Getting to know the problem

Although this is a hugely useful and valuable feature, we got the impression that not enough bugs were getting linked. The problem that I wanted my team to solve was to explore why these bugs were not being linked and help to increase the linkages. Although we had a hunch that not that many people were linking bugs, we really had no idea. We had no concrete statistics to back up our assumptions. Our first task was to learn more about the problem.

To do this we produced a tool called the Ubuntu Upstream Report (https://launchpad.net/ubuntu/+upstreamreport). The report mined the Launchpad bug tracker to show the top 100 upstreams shipped in Ubuntu ordered by the greatest number of open bugs. We knew that these 100 upstreams were likely to be the largest and most significant projects: more bugs in the open source world typically means larger projects with more users using the software and therefore filing more bugs.

For each upstream in the report, we showed which bugs were open, which had been triaged to determine they were actually bugs, which were known to be upstream bugs (marked as upstream tasks), and which had linkages associated with those tasks.

The differences between the numbers generated interesting conclusions. As an example, if a project had a significant difference between the number of open bugs and the number of triaged bugs, we knew which projects needed more help with bug triage.

A key conclusion that we discovered was the difference in many projects between the upstream tasks and linked bugs numbers. As an example, one project had 229 bugs with upstream tasks but only 101 of those bugs had upstream links. This left a total of 128 bugs that were known to be upstream bugs that didn’t have linkages. This told us that the project was not linking bugs well and needed help. It also told us which specific bugs were known to be upstream bugs without linkages. We could use these bugs as a target for our community to make those linkages.

Breaking down the conversation

The raw data in the upstream report was not only the foundation of exploring the problem, but also solving it. The data confirmed our suspicion that a lot of bugs weren’t being linked. On a practical level, it helped us to understand which upstreams needed help, and therefore which contributors to speak to.

To raise the number of linked bugs, we needed to identify the three key aspects of the bughandling process (reporting, triage, and linking) and improve them. Our first area to focus on was linking, as it was the most immediate goal. We examined and optimized each step in the process of linking a bug. We discovered, for example, that finding bugs in the upstream bug tracker took the most time, so we explored methods of syncing bug data across the Launchpad and upstream bug reports automatically, automating duplication searches, and simplifying the user interface for the whole shebang.

Our next step was to optimize the reporting and triage process. This was an area in which we had already poured extensive work, and our workflow here was generally pretty good. Where there appeared to be a disconnect, though, was with documentation and people reading and making use of that documented best practice. Although there was plenty of documentation, it was scattered all over the Ubuntu wiki, difficult to navigate, and complex to read. We reorganized and improved it, dividing it into better sections and making it easier to understand. We organized documentation days; encouraged community participation; celebrated the improvements; and encouraged our bug triagers to review, expand, and make use of the new documentation that was produced.