Ubuntu Live Testing

Introduction

Context

Given the ever-increasing adoption of Ubuntu and its flavors and, thus, also of packages for it, the growing challenge for QA-Related teams demands new, more effective and smarter strategies.
Traditional approaches, such as simple ISO-Testing done by community members must now give place to new initiatives, such as automated testing, specialized testing communities and teams, new tools, methods and resources.
In parallel, while Ubuntu users are encouraged to reports bugs via apport and Launchpad, the number of reports is currently overwhelming, as is the growth in the amount of reports. Considering most users are not technical-oriented, a large portion of reports are invalid. Even reports that refer to existing bugs are, frequently, incomplete. Developers not only need to browse through many reports on Launchpad, but also have to filter reports that have high value-added and precise information. As a result, some bugs persist within a release life-cycle and, also, between releases.

Ubuntu Previous Attempts

• Discuss This: https://wiki.ubuntu.com/Testing/UbuntuTestingDay -- effenberg0x0 2012-04-05 06:43:01

• Discuss this: https://wiki.ubuntu.com/UbuntuBugDay -- effenberg0x0 2012-04-05 06:43:01

Fedora: Testing Days

Fedora has implemented an interesting strategy named "Testing Days". It consists of a simple web page defining:

• What package will be tested in a specific date;
• Who will be available for supporting the session (helpers, experienced testers, developers);
• The prerequisites to join the testing session (packages versions, tools,etc);
• Links to download the target package (to be tested) as well as related packages and tools;
• Instructions on how to install the target package and related packages and tools;
• Planned testing:
  • Previously developed testcases;
• Unplanned testing:
  • Contributed by the community, frequently during the testing session;
• Links and instructions on how to report a bug;

• A moderated IRC channel is used by participants of the Testing Day. Developers join this channel and actively interact with testers, providing fixes, test builds, workarounds, testing instructions, etc.

As a result of such strategy, special focus is given to specific packages (generally critical ones), putting together developers, the community, testers and helpers. Known and previously unknown bugs are fixed or, at least, confirmed at these sessions. Even when bugs are not immediately fixed, developers receive large amounts of information, which is used later on the bug fix procedures.

Implementation in Ubuntu: Ubuntu Live Testing

Requirements:
Ubuntu has a large community, strong leadership and is capable of creating awareness over it's events through it's many media. It can gather interested testers and users for such sessions and has the technical resources to host them.
The following is a preliminary list of basic requirements needed to further develop this project. Each item is analyzed in-depth in the sections below.

• Communication to the community: Gather support and involvement;

• Technical resources: Moderated IRC channel, pad.ubuntu.com, testing repo, testing ISO, session bug tracker;

• In-depth Analysis: Selection of bugs and packages;

• Content: Omnibus wiki page, up-to-date, easy and correct tutorials;

• Staff and support: To develop/support items above, plus testers and developers;

Communication to the community

• UbuntuForums, it's staff and members, sub-forum communities and its leaders;

• IRC, Ubuntu-related channels, including not only those targeted at technical users but also community-oriented channels;
• Ubuntu QA Team
• Other QA-Centric communities and teams;

• AskUbuntu;

• Ubuntu news websites;

Partnership: UbuntuForums

UbuntuForums, as the most successful, enduring and large online community of Ubuntu users endorses this project through its Forum Council. This project was ignited by BodhiZazen, a leader of this council. UbuntuForums will collaborate by:

• Publicizing Ubuntu Live Testing sessions and inviting the community to join;
• Informing the forum staff about the event and inviting them to join;

Technical resources

The only technical requirements are:

• A web page, containing all the information related to the session (target package, instructions, etc) must be set previously to the event;
  • A Ubuntu Wiki page can be set with the required information;
  • A page withing the QA subdomain can also be used;
• A moderated IRC channel must be defined and used to host real-time interaction between users, testers and developers;
  • Ubuntu IRC staff already provides all the structure needed to host moderated sessions as well as helpers and IRC staff to support users;
  • Ubuntu IRC sessions are automatically logged and posted to the web, which can help developers and testers as a base for informations and discussions held at the session;

Analysis, selection of bugs and packages

By using the current QA structure, and information sources such as the Ubuntu QA Reports, we can easily define a list of target packages, which are critical to the Operating System and, yet, have a significant amount of persisting bugs.

Staff and supporters

• General moderation will be done by the Ubuntu IRC staff;
• QATeam, its sub-teams and QA-Related communities will be invited to join the sessions;

• UbuntuForum staff and members will be invited to participate;

• Community members will be invited to join at UbuntuForums;

• Other Ubuntu communities will be free to invite their members;

Testers

• A small group (5?) of testers will he selected by this team for the trial session;
• Define how testers will be selected for upcoming events;
• Define if the event will be opened to the community or just to selected testers;

Bringing in developers

• The Ubuntu QA Community Coordinator, with the support of QA-related teams, will contact developers, based on the packages selected for the sessions;
• And open invitation for developers will be published to the appropriate mailing lists. Developers that would like to have their packages tested in Live Testing Sessions this project coordination to schedule a date;

Trial: A private Live Testing session

• In order to evaluate the effectiveness, problems, strong and week points on this strategy, a trial session will be developed;
• Implementation difficulties, results and other aspects will be evaluated / fixed by the team before a public set of events is announced;

Trial implementation processes and delegation

"Trial" Live Testing Session Planning

Endurance and regularity: How to make this strategy sustainable in long-term?

• Previous strategies have failed in Ubuntu;
• Other similar activities have also been discontinued or abandoned;
• Why? Failure to get results (bugs fixed)? No interest from developers, testers? Failure to communicate and create awareness over the initiative? Poor and/or non-functional technical resources? Documentation?

Next steps

Primary

• Team meeting to evaluate this doc (April, 4th, 2100UTC) - OK;

• Next Team meeting: Mature strategy and delegate tasks (April, 11th, 2100UTC);

• Plan and host trial session - ?

Secondary

• Next Team meeting: Evaluate results, improve process and delegate new tasks (April, 18th, 2100UTC);

• Communicate Ubuntu Governance Boards and obtain approval;
• Request support from IRC Team leadership;
• Inform QA-Related Teams and confirm support for upcoming sessions;
• Develop and send "open invitation" to developers, via proper channels;
• Evaluate potential packages and contact developers;
• Define dates;
• Publicize to community;

Agenda - Next Team Meetings

Insert topics directly in the bullets below or mail Effenberg0x0 [at] Ubuntu.com if you'd like me to do it. Please sign your bullets with @SIG@ at the end. It automatically adds your name and timestamp to your comments (when the page is saved, not when previewed), so we can keep track of who said what and when.

Apr. 15th. 2012

Apr. 11th. 2012

• Discuss previous attempts, linked at the topic "Ubuntu Previous Attempts" in this document (see index). They were completely unknown to me. Why did they fail? What can we learn from them? -- effenberg0x0 2012-04-05 06:59:22

• Infrastructure: A more "in-depth" look at Fedora's implementation revealed to me some infrastructure we currently do not have: Special repos for Live Testing ISOs, Testing Builds of packages updated during the session, a tracker updated in real time during the session, with testing results of session participants, a system to host each tester hardware profile (relevant for developers to evaluate each test result vs hardware tested). -- effenberg0x0 2012-04-05 06:59:22

• Mature strategy and delegate tasks -- effenberg0x0 2012-04-06 23:52:38

Apr. 4th. 2012 (kickoff)

Topics discussed at this meeting are being summarized and will be inserted here by Apr. 9th, 2012 (Monday) -- effenberg0x0 2012-04-05 06:43:01 References:

• http://meetbot.fedoraproject.org/fedora-bugzappers/2012-03-02/f17-beta-blocker-review-1.2012-03-02-17.00.html

• https://bugzilla.redhat.com/show_bug.cgi?id=752650

• http://fedoraproject.org/wiki/Blocker_Bug_FAQ

• http://meetbot.fedoraproject.org/fedora-bugzappers/2012-03-02/f17-beta-blocker-review-1.2012-03-02-17.00.html

• http://reports.qa.ubuntu.com/reports/kernel-bugs/reports/rls-mgr-p-tracking-bugs.html

• https://bugs.launchpad.net/ubuntu/+source/linux/+bug/899244

• https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=785053I

Brainstorming

Please sign your bullets with @SIG@ at the end. It automatically adds your name and timestamp to your comments (when the page is saved, not when previewed), so we can keep track of who said what and when.

Contact the project coordination

Project Management

Project Supporters