LiveTesting

Differences between revisions 1 and 9 (spanning 8 versions)
Revision 1 as of 2012-04-07 00:07:46
Size: 46
Editor: 189-38-251-58
Comment:
Revision 9 as of 2012-04-10 16:39:02
Size: 14810
Editor: 189
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
<<include(U+1/OngoingProjects/LiveTesting)>> <<Include(LiveTesting/Resources/NavBar)>>
||<tablestyle="float:right; font-size: 0.9em; width:35%; background:#F1F1DD; margin: 2.5em 0 1em 1em; border: none; border-radius: 10px 10px 10px 10px;" style="margin: 0; padding:0.5em;"><<TableOfContents(6)>>||
----
= Ubuntu Live Testing =
== Introduction ==
=== Context ===
Given the ever-increasing adoption of Ubuntu and its flavors and, thus, also of packages for it, the growing challenge for QA-Related teams demands new, more effective and smarter strategies.<<BR>>
Traditional approaches, such as simple ISO-Testing done by community members must now give place to new initiatives, such as automated testing, specialized testing communities and teams, new tools, methods and resources.<<BR>>
In parallel, while Ubuntu users are encouraged to reports bugs via [[https://wiki.ubuntu.com/Apport|apport]] and [[https://launchpad.net|Launchpad]], the number of reports is currently overwhelming, as is the growth in the amount of reports. Considering most users are not technical-oriented, a large portion of reports are invalid. Even reports that refer to existing bugs are, frequently, incomplete.
Developers not only need to browse through many reports on Launchpad, but also have to filter reports that have high value-added and precise information. As a result, some bugs persist within a release life-cycle and, also, between releases.<<BR>>
=== Ubuntu Previous Attempts ===
 * Discuss This: https://wiki.ubuntu.com/Testing/UbuntuTestingDay -- [[LaunchpadHome:effenberg0x0]] <<DateTime(2012-04-05T06:43:01Z)>>
 * Discuss this: https://wiki.ubuntu.com/UbuntuBugDay -- [[LaunchpadHome:effenberg0x0]] <<DateTime(2012-04-05T06:43:01Z)>>
=== Fedora: Testing Days ===
Fedora has implemented an interesting strategy named '''"Testing Days"'''. It consists of a simple web page defining:
 * What package will be tested in a specific date;
 * Who will be available for supporting the session (helpers, experienced testers, developers);
 * The prerequisites to join the testing session (packages versions, tools,etc);
 * Links to download the target package (to be tested) as well as related packages and tools;
 * Instructions on how to install the target package and related packages and tools;
 * Planned testing:
  * Previously developed testcases;
 * Unplanned testing:
  * Contributed by the community, frequently during the testing session;
 * Links and instructions on how to report a bug;
 * A moderated IRC channel is used by participants of the '''Testing Day'''. Developers join this channel and actively interact with testers, providing fixes, test builds, workarounds, testing instructions, etc.

As a result of such strategy, special focus is given to specific packages (generally critical ones), putting together developers, the community, testers and helpers. Known and previously unknown bugs are fixed or, at least, confirmed at these sessions. Even when bugs are not immediately fixed, developers receive large amounts of information, which is used later on the bug fix procedures.
== Implementation in Ubuntu: Ubuntu Live Testing ==
'''Requirements:'''<<BR>>
Ubuntu has a large community, strong leadership and is capable of creating awareness over it's events through it's many media. It can gather interested testers and users for such sessions and has the technical resources to host them.<<BR>>
The following is a preliminary list of basic requirements needed to further develop this project. Each item is analyzed in-depth in the sections below.<<BR>>
 * '''Communication to the community:''' Gather support and involvement;
 * '''Technical resources:''' Moderated IRC channel, pad.ubuntu.com, testing repo, testing ISO, session bug tracker;
 * '''In-depth Analysis:''' Selection of bugs and packages;
 * '''Content:''' Omnibus wiki page, up-to-date, easy and correct tutorials;
 * '''Staff and support:''' To develop/support items above, plus '''testers''' and '''developers''';
=== Communication to the community ===
 * UbuntuForums, it's staff and members, sub-forum communities and its leaders;
 * IRC, Ubuntu-related channels, including not only those targeted at technical users but also community-oriented channels;
 * Ubuntu QA Team
 * Other QA-Centric communities and teams;
 * Ask``U``buntu;
 * Ubuntu news websites;
==== Partnership: UbuntuForums ====
UbuntuForums, as the most successful, enduring and large online community of Ubuntu users endorses this project through its [[https://wiki.ubuntu.com/ForumCouncil|Forum Council]]. This project was ignited by [[https://wiki.ubuntu.com/BodhiZazen|BodhiZazen]], a leader of this council. UbuntuForums will collaborate by:
 * Publicizing Ubuntu Live Testing sessions and inviting the community to join;
 * Informing the forum staff about the event and inviting them to join;
=== Technical resources ===
The only technical requirements are:
 * A web page, containing all the information related to the session (target package, instructions, etc) must be set previously to the event;
  * A Ubuntu Wiki page can be set with the required information;
  * A page withing the QA subdomain can also be used;
 * A moderated IRC channel must be defined and used to host real-time interaction between users, testers and developers;
  * Ubuntu IRC staff already provides all the structure needed to host moderated sessions as well as helpers and IRC staff to support users;
  * Ubuntu IRC sessions are automatically logged and posted to the web, which can help developers and testers as a base for informations and discussions held at the session;
=== Analysis, selection of bugs and packages ===
By using the current QA structure, and information sources such as the [[http://reports.qa.ubuntu.com/reports/|Ubuntu QA Reports]], we can easily define a list of target packages, which are critical to the Operating System and, yet, have a significant amount of persisting bugs.
=== Staff and supporters ===
 * General moderation will be done by the Ubuntu IRC staff;
 * QATeam, its sub-teams and QA-Related communities will be invited to join the sessions;
 * UbuntuForum staff and members will be invited to participate;
 * Community members will be invited to join at UbuntuForums;
 * Other Ubuntu communities will be free to invite their members;
==== Testers ====
 * A small group (5?) of testers will he selected by this team for the trial session;
 * Define how testers will be selected for upcoming events;
 * Define if the event will be opened to the community or just to selected testers;
==== Bringing in developers ====
 * The Ubuntu QA Community Coordinator, with the support of QA-related teams, will contact developers, based on the packages selected for the sessions;
 * And open invitation for developers will be published to the appropriate mailing lists. Developers that would like to have their packages tested in Live Testing Sessions this project coordination to schedule a date;
=== Trial: A private Live Testing session ===
 * In order to evaluate the effectiveness, problems, strong and week points on this strategy, a trial session will be developed;
 * Implementation difficulties, results and other aspects will be evaluated / fixed by the team before a public set of events is announced;
==== Trial implementation processes and delegation ====
'''"Trial" Live Testing Session Planning'''
||<tablestyle="float:center; width:100%; background:#F1F1DD; color: #000000; margin: 0; padding: 0; border: 0; text-align: left;" rowstyle="margin: 0; padding: 0; border: 0; background: #3C3B37; color: #FFFFFF;" style="text-align: center; width:20%;">~+'''Ativitity'''+~||<style="text-align: center; width:20%;">~+'''Description'''+~||<style="text-align: center; width:30%;">~+'''Delegated to'''+~||<style="text-align: center; width:20%;">~+'''Deadline'''+~||<style="text-align: center; width:30%;">~+'''Status'''+~||
||IRC Infrastructure||Get temporary IRC infrastructure and moderation staff for a single trial event.||?||?||?||
||Testing Repo||A testing repository must be set exclusively for this session. The developers must have the rights to upload to it in real time during the session.||?||?||?||
||ISO||A previously developed ISO must be made, to be used by testers during the session||?||?||?||
||Tracker||A Tracker must be accessible by the developer and testers during the session. Results must continue to be available after the session.||?||?||?||
||Analysis/Selection||Select bugs and corresponding packages, reach their developers to invite for the trial session||?||?||?||
||Content||Create omnibus wiki page to support the session. Link, develop or improve related tutorials. Evaluate how much time it would take to have such a content ready for every session.||?||?||?||
||Testers||Select small group (5?) of testers, to participate in the trial session.||?||?||?||
=== Endurance and regularity: How to make this strategy sustainable in long-term? ===
 * Previous strategies have failed in Ubuntu;
 * Other similar activities have also been discontinued or abandoned;
 * Why? Failure to get results (bugs fixed)? No interest from developers, testers? Failure to communicate and create awareness over the initiative? Poor and/or non-functional technical resources? Documentation?
== Next steps ==
'''Primary'''
 * Team meeting to evaluate this doc (April, 4th, 2100UTC) - '''OK''';
 * '''Next Team meeting''': Mature strategy and delegate tasks (April, 11th, 2100UTC);
 * Plan and host trial session - ?
'''Secondary'''
 * '''Next Team meeting''': Evaluate results, improve process and delegate new tasks (April, 18th, 2100UTC);
 * Communicate Ubuntu Governance Boards and obtain approval;
 * Request support from IRC Team leadership;
 * Inform QA-Related Teams and confirm support for upcoming sessions;
 * Develop and send "open invitation" to developers, via proper channels;
 * Evaluate potential packages and contact developers;
 * Define dates;
 * Publicize to community;
== Agenda - Next Team Meetings ==
Insert topics directly in the bullets below or mail Effenberg0x0 [at] Ubuntu.com if you'd like me to do it. '''Please sign your bullets with ``@``SIG``@`` at the end. It automatically adds your name and timestamp to your comments (when the page is saved, not when previewed), so we can keep track of who said what and when.'''
=== Apr. 15th. 2012 ===
 *
 *
 *
 *
 *
 *
=== Apr. 11th. 2012 ===
 * Discuss '''previous attempts''', linked at the topic '''"Ubuntu Previous Attempts"''' in this document (see index). They were completely unknown to me. Why did they fail? What can we learn from them? -- [[LaunchpadHome:effenberg0x0]] <<DateTime(2012-04-05T06:59:22Z)>>
 * Infrastructure: A more "in-depth" look at Fedora's implementation revealed to me some infrastructure we currently do not have: Special repos for Live Testing ISOs, Testing Builds of packages updated during the session, a tracker updated in real time during the session, with testing results of session participants, a system to host each tester hardware profile (relevant for developers to evaluate each test result vs hardware tested). -- [[LaunchpadHome:effenberg0x0]] <<DateTime(2012-04-05T06:59:22Z)>>
 * Mature strategy and delegate tasks -- [[LaunchpadHome:effenberg0x0]] <<DateTime(2012-04-06T23:52:38Z)>>
 *
 *
 *
 *
=== Apr. 4th. 2012 (kickoff) ===
Topics discussed at this meeting are being summarized and will be inserted here by Apr. 9th, 2012 (Monday) -- [[LaunchpadHome:effenberg0x0]] <<DateTime(2012-04-05T06:43:01Z)>>
'''References:'''<<BR>>
 * http://meetbot.fedoraproject.org/fedora-bugzappers/2012-03-02/f17-beta-blocker-review-1.2012-03-02-17.00.html
 * https://bugzilla.redhat.com/show_bug.cgi?id=752650
 * http://fedoraproject.org/wiki/Blocker_Bug_FAQ
 * http://meetbot.fedoraproject.org/fedora-bugzappers/2012-03-02/f17-beta-blocker-review-1.2012-03-02-17.00.html
 * http://reports.qa.ubuntu.com/reports/kernel-bugs/reports/rls-mgr-p-tracking-bugs.html
 * https://bugs.launchpad.net/ubuntu/+source/linux/+bug/899244
 * https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=785053I
== Brainstorming ==
'''Please sign your bullets with ``@``SIG``@`` at the end. It automatically adds your name and timestamp to your comments (when the page is saved, not when previewed), so we can keep track of who said what and when.'''
 *
 *
 *
 *
 *
 *
 *
 *
== Contact the project coordination ==
'''Project Management'''
||<tablestyle="float:center; width:100%; background:#F1F1DD; color: #000000; margin: 0; padding: 0; border: 0; text-align: left;" rowstyle="margin: 0; padding: 0; border: 0; background: #3C3B37; color: #FFFFFF;" style="text-align: center; width:20%;">~+'''UF Handle'''+~||<style="text-align: center; width: 20%;">~+'''LP Handle'''+~||<style="text-align: center; width: 30%;">~+'''Name'''+~||<style="text-align: center; width: 20%;">~+'''Team'''+~||<style="text-align: center;width: 30%;">~+'''Role'''+~||
||[[http://ubuntuforums.org/member.php?u=89054|BodhiZazen]]||[[https://launchpad.net/~bodhi.zazen|BodhiZazen]]||Mr. Bo D'Zazen III||[[https://wiki.ubuntu.com/ForumCouncil|Forum Council]]||General Management||
<<BR>><<BR>>
'''Project Supporters'''
||<tablestyle="float:center; width:100%; background:#F1F1DD; color: #000000; margin: 0; padding: 0; border: 0; text-align: left;" rowstyle="margin: 0; padding: 0; border: 0; background: #3C3B37; color: #FFFFFF;" style="text-align: center; width:20%;">~+'''UF Handle'''+~||<style="text-align: center; width:20%;">~+'''LP Handle'''+~||<style="text-align: center; width:30%;">~+'''Name'''+~||<style="text-align: center; width:20%;">~+'''Team'''+~||<style="text-align: center; width:30%;">~+'''Role'''+~||
||[[http://ubuntuforums.org/member.php?u=92967|Guitara]]||[[hhttps://launchpad.net/~nskaggs|nskaggs]]||Nicholas Skaggs||[[https://wiki.ubuntu.com/QATeam|Canonical Community QA Coordinator]]||Team Undercover Operations Manager||
||[[http://ubuntuforums.org/member.php?u=77104|Cariboo907]]||[[https://launchpad.net/~cariboo907|Cariboo907]]||Jim Kielman||[[https://wiki.ubuntu.com/ForumCouncil|Forum Council]]||Team Tactical Driver||
||[[http://ubuntuforums.org/member.php?u=129087|Coffeecat]]||[[https://launchpad.net/~coffeecat|Coffeecat]]||Coffeecat||[[https://wiki.ubuntu.com/ForumCouncil|Forum Council]]||FEBS - Feline Energetic Beverages Specialist||
||[[http://ubuntuforums.org/member.php?u=260143|Effenberg0x0]]||[[https://launchpad.net/~effenberg0x0|Effenberg0x0]]||Alvaro Leal||[[https://wiki.ubuntu.com/U+1|U+1]]||Team Janitor||
||[[http://ubuntuforums.org/member.php?u=1087323|Grahammechanical]]||[[https://launchpad.net/~lucking|lucking]]||Graham Lucking||[[https://wiki.ubuntu.com/U+1|U+1]]||Team Explosives Specialist||

About
IconsPage/picto_articles_48.png

Home
About the Project

DocsIconsPage/picto_articles_48.png

Project Specifications
Meeting Reports, ToDo and Agenda

Live TestingIconsPage/picto_articles_48.png

Sessions Schedule
Sessions Reports

ContactIconsPage/picto_articles_48.png

Contact the Team

Click here to access full instructions for the current Live Testing Session.


Ubuntu Live Testing

Introduction

Context

Given the ever-increasing adoption of Ubuntu and its flavors and, thus, also of packages for it, the growing challenge for QA-Related teams demands new, more effective and smarter strategies.
Traditional approaches, such as simple ISO-Testing done by community members must now give place to new initiatives, such as automated testing, specialized testing communities and teams, new tools, methods and resources.
In parallel, while Ubuntu users are encouraged to reports bugs via apport and Launchpad, the number of reports is currently overwhelming, as is the growth in the amount of reports. Considering most users are not technical-oriented, a large portion of reports are invalid. Even reports that refer to existing bugs are, frequently, incomplete. Developers not only need to browse through many reports on Launchpad, but also have to filter reports that have high value-added and precise information. As a result, some bugs persist within a release life-cycle and, also, between releases.

Ubuntu Previous Attempts

Fedora: Testing Days

Fedora has implemented an interesting strategy named "Testing Days". It consists of a simple web page defining:

  • What package will be tested in a specific date;
  • Who will be available for supporting the session (helpers, experienced testers, developers);
  • The prerequisites to join the testing session (packages versions, tools,etc);
  • Links to download the target package (to be tested) as well as related packages and tools;
  • Instructions on how to install the target package and related packages and tools;
  • Planned testing:
    • Previously developed testcases;
  • Unplanned testing:
    • Contributed by the community, frequently during the testing session;
  • Links and instructions on how to report a bug;
  • A moderated IRC channel is used by participants of the Testing Day. Developers join this channel and actively interact with testers, providing fixes, test builds, workarounds, testing instructions, etc.

As a result of such strategy, special focus is given to specific packages (generally critical ones), putting together developers, the community, testers and helpers. Known and previously unknown bugs are fixed or, at least, confirmed at these sessions. Even when bugs are not immediately fixed, developers receive large amounts of information, which is used later on the bug fix procedures.

Implementation in Ubuntu: Ubuntu Live Testing

Requirements:
Ubuntu has a large community, strong leadership and is capable of creating awareness over it's events through it's many media. It can gather interested testers and users for such sessions and has the technical resources to host them.
The following is a preliminary list of basic requirements needed to further develop this project. Each item is analyzed in-depth in the sections below.

  • Communication to the community: Gather support and involvement;

  • Technical resources: Moderated IRC channel, pad.ubuntu.com, testing repo, testing ISO, session bug tracker;

  • In-depth Analysis: Selection of bugs and packages;

  • Content: Omnibus wiki page, up-to-date, easy and correct tutorials;

  • Staff and support: To develop/support items above, plus testers and developers;

Communication to the community

  • UbuntuForums, it's staff and members, sub-forum communities and its leaders;

  • IRC, Ubuntu-related channels, including not only those targeted at technical users but also community-oriented channels;
  • Ubuntu QA Team
  • Other QA-Centric communities and teams;
  • AskUbuntu;

  • Ubuntu news websites;

Partnership: UbuntuForums

UbuntuForums, as the most successful, enduring and large online community of Ubuntu users endorses this project through its Forum Council. This project was ignited by BodhiZazen, a leader of this council. UbuntuForums will collaborate by:

  • Publicizing Ubuntu Live Testing sessions and inviting the community to join;
  • Informing the forum staff about the event and inviting them to join;

Technical resources

The only technical requirements are:

  • A web page, containing all the information related to the session (target package, instructions, etc) must be set previously to the event;
    • A Ubuntu Wiki page can be set with the required information;
    • A page withing the QA subdomain can also be used;
  • A moderated IRC channel must be defined and used to host real-time interaction between users, testers and developers;
    • Ubuntu IRC staff already provides all the structure needed to host moderated sessions as well as helpers and IRC staff to support users;
    • Ubuntu IRC sessions are automatically logged and posted to the web, which can help developers and testers as a base for informations and discussions held at the session;

Analysis, selection of bugs and packages

By using the current QA structure, and information sources such as the Ubuntu QA Reports, we can easily define a list of target packages, which are critical to the Operating System and, yet, have a significant amount of persisting bugs.

Staff and supporters

  • General moderation will be done by the Ubuntu IRC staff;
  • QATeam, its sub-teams and QA-Related communities will be invited to join the sessions;
  • UbuntuForum staff and members will be invited to participate;

  • Community members will be invited to join at UbuntuForums;

  • Other Ubuntu communities will be free to invite their members;

Testers

  • A small group (5?) of testers will he selected by this team for the trial session;
  • Define how testers will be selected for upcoming events;
  • Define if the event will be opened to the community or just to selected testers;

Bringing in developers

  • The Ubuntu QA Community Coordinator, with the support of QA-related teams, will contact developers, based on the packages selected for the sessions;
  • And open invitation for developers will be published to the appropriate mailing lists. Developers that would like to have their packages tested in Live Testing Sessions this project coordination to schedule a date;

Trial: A private Live Testing session

  • In order to evaluate the effectiveness, problems, strong and week points on this strategy, a trial session will be developed;
  • Implementation difficulties, results and other aspects will be evaluated / fixed by the team before a public set of events is announced;

Trial implementation processes and delegation

"Trial" Live Testing Session Planning

Ativitity

Description

Delegated to

Deadline

Status

IRC Infrastructure

Get temporary IRC infrastructure and moderation staff for a single trial event.

?

?

?

Testing Repo

A testing repository must be set exclusively for this session. The developers must have the rights to upload to it in real time during the session.

?

?

?

ISO

A previously developed ISO must be made, to be used by testers during the session

?

?

?

Tracker

A Tracker must be accessible by the developer and testers during the session. Results must continue to be available after the session.

?

?

?

Analysis/Selection

Select bugs and corresponding packages, reach their developers to invite for the trial session

?

?

?

Content

Create omnibus wiki page to support the session. Link, develop or improve related tutorials. Evaluate how much time it would take to have such a content ready for every session.

?

?

?

Testers

Select small group (5?) of testers, to participate in the trial session.

?

?

?

Endurance and regularity: How to make this strategy sustainable in long-term?

  • Previous strategies have failed in Ubuntu;
  • Other similar activities have also been discontinued or abandoned;
  • Why? Failure to get results (bugs fixed)? No interest from developers, testers? Failure to communicate and create awareness over the initiative? Poor and/or non-functional technical resources? Documentation?

Next steps

Primary

  • Team meeting to evaluate this doc (April, 4th, 2100UTC) - OK;

  • Next Team meeting: Mature strategy and delegate tasks (April, 11th, 2100UTC);

  • Plan and host trial session - ?

Secondary

  • Next Team meeting: Evaluate results, improve process and delegate new tasks (April, 18th, 2100UTC);

  • Communicate Ubuntu Governance Boards and obtain approval;
  • Request support from IRC Team leadership;
  • Inform QA-Related Teams and confirm support for upcoming sessions;
  • Develop and send "open invitation" to developers, via proper channels;
  • Evaluate potential packages and contact developers;
  • Define dates;
  • Publicize to community;

Agenda - Next Team Meetings

Insert topics directly in the bullets below or mail Effenberg0x0 [at] Ubuntu.com if you'd like me to do it. Please sign your bullets with @SIG@ at the end. It automatically adds your name and timestamp to your comments (when the page is saved, not when previewed), so we can keep track of who said what and when.

Apr. 15th. 2012

Apr. 11th. 2012

  • Discuss previous attempts, linked at the topic "Ubuntu Previous Attempts" in this document (see index). They were completely unknown to me. Why did they fail? What can we learn from them? -- effenberg0x0 2012-04-05 06:59:22

  • Infrastructure: A more "in-depth" look at Fedora's implementation revealed to me some infrastructure we currently do not have: Special repos for Live Testing ISOs, Testing Builds of packages updated during the session, a tracker updated in real time during the session, with testing results of session participants, a system to host each tester hardware profile (relevant for developers to evaluate each test result vs hardware tested). -- effenberg0x0 2012-04-05 06:59:22

  • Mature strategy and delegate tasks -- effenberg0x0 2012-04-06 23:52:38

Apr. 4th. 2012 (kickoff)

Topics discussed at this meeting are being summarized and will be inserted here by Apr. 9th, 2012 (Monday) -- effenberg0x0 2012-04-05 06:43:01 References:

Brainstorming

Please sign your bullets with @SIG@ at the end. It automatically adds your name and timestamp to your comments (when the page is saved, not when previewed), so we can keep track of who said what and when.

Contact the project coordination

Project Management

UF Handle

LP Handle

Name

Team

Role

BodhiZazen

BodhiZazen

Mr. Bo D'Zazen III

Forum Council

General Management



Project Supporters

UF Handle

LP Handle

Name

Team

Role

Guitara

nskaggs

Nicholas Skaggs

Canonical Community QA Coordinator

Team Undercover Operations Manager

Cariboo907

Cariboo907

Jim Kielman

Forum Council

Team Tactical Driver

Coffeecat

Coffeecat

Coffeecat

Forum Council

FEBS - Feline Energetic Beverages Specialist

Effenberg0x0

Effenberg0x0

Alvaro Leal

U+1

Team Janitor

Grahammechanical

lucking

Graham Lucking

U+1

Team Explosives Specialist

LiveTesting (last edited 2012-04-10 16:39:02 by 189)