Feedback on the Xorg Proprietary Drivers Weekly Program

Summary

During the Lucid development cycle we run a testing program for proprietary graphic drivers and their integration with Ubuntu. We tried testing instructions to be as clear as possible and having an easy way to report results. We used the xorg instance of the ISO tracker.

The aim of the program was testing the same thing every week, to catch possible regressions as soon as possible.

Feedback from participants

The general feedback from participants (testers and developers) was very positive. What people most liked about the project is that having a defined scope makes it easier for a newbie to start helping. Also, it improves the quality of the tests.

All the participants that replied to the Call For Feedback are willing to participate in similar programs in the next cycles. Also, four of them are willing to help organizing this kind of efforts, not only on X testing, but also other packages.

Program improvements

• Better communication between testers and developers because, although developers were looking weekly to the results, some testers had the impression that developers were not taking care of results.
• Better communication on "expected failures", indicating that, if the test magically passed, what kind of debuggin information would be helpful to the developers.
• A way to indicate regressions to previous passing tests. Also, there should be some mechanism to escalate discoveries of serious regressions, that doesn't depend on someone remembering to look at the web page each week (i.e. an automated email to @ubuntu-devel)
• Better documentation (screenshots?)
• Automation (python scripts on go/no-go). Autoinstall of Blender and testcases.

Improvements to the ISO tracker

• Report history by tester. (good for weekly programs)
• Historical bugs on a particular test cases (known issues)
• Possibility to flag tests as regressions.
• After logging in, take testers back to the page they were on.
• Allow either listing test case report pages in the same order as they are in the instructions, or show the test case instructions on the page for inputting the results.
• Allow tracking results from multiple machines from one tester account.