StructuredTestingPrograms

Summary

uring the Lucid development cycle we run a testing program for proprietary graphic drivers and their integration with Ubuntu. Details of the program can be found at the wiki https://wiki.ubuntu.com/X/Testing/ProprietaryDrivers/WeeklyProgram.

Feedback from participants https://wiki.ubuntu.com/X/Testing/ProprietaryDrivers/WeeklyProgram/Feedback was very positive and some of them are even willing to organize similar testing efforts.

We will gather feedback to improve this kind of initiatives.

Release Note

This section should include a paragraph describing the end-user impact of this change. It is meant to be included in the release notes of the first release in which it is implemented. (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.)

It is mandatory.

Rationale

This should cover the _why_: why is this change being proposed, what justifies it, where we see this justified.

User stories

Assumptions

Design

You can have subsections that better describe specific parts of the issue.

Implementation

This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:

UI Changes

Should cover changes required to the UI, or specific UI that is required to implement this

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Migration

Include:

  • data migration, if any
  • redirects from old URLs to new ones, if any
  • how users will be pointed to the new way of doing things, if necessary.

Test/Demo Plan

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

BoF agenda and discussion

Agenda

  • Explanation of the project to those who didn't know about it
  • Infrastructure
    • The ISO tracker: what worked, what didn't and what's missing
    • The testcases wiki
  • The design team use case format. How can we use it?
  • Communication
    • Mailing list
    • Forums
    • Documentation
  • Community engagement

See also RedHat's Bodhi - http://fedoraproject.org/wiki/Bodhi_Guide - process which utilizes tester community to test packages before they're put into the distro officially (analogous to our *-proposed and SRU processes, but which includes structured testing and package testing karma).

Discussion

# Explanation of the project to those who didn't know about it

Structured testing, executing test cases throughout the development cycle
https://wiki.ubuntu.com/X/Testing/ProprietaryDrivers/WeeklyProgram

Daily reports
http://qa.ubuntu.com/reports/xorg_prop_drivers/

Coordinated through Launchpad team and mailing list

# Infrastructure

    * The ISO tracker: what worked, what didn't and what's missing
    * The testcases wiki 

# The design team use case format. How can we use it?
Output of design use case is acceptance criteria
Have a single workflow starting with design team's use cases, flowing through to development and then QA

They also have a Google Docs-based tool for editing and viewing use cases

# Communication

People found that having the opportunity to communicate with the devs was a very positive experience. Responsive developers-tester communication is a clear benefit of working with testcases rather than reporting bugs and waiting for feedback.

Need to keep the simplicity of the testcases in order to make it easy for people to start testing and maintain focus

    * Mailing list
    * Forums
    * Documentation 

# Community engagement 


Problem: reporting
 - Overlap of tests being done on qa.ubuntu.com and checkbox tests - many listed here: http://testcases.qa.ubuntu.com/Hardware - avoiding duplication by having checkbox submit info to the QA tracker.
 - Check box has modular reporting - a plugin could be written to report to additional locations (current launchpad or certification website)
 - Assembly-line type testing which includes design teams use case mapper, and QA testcase tracker. 
 - Use of project Sikuli can assist with automation of tests - http://groups.csail.mit.edu/uid/sikuli/

 - Different information

Either an individual or checkbox needs to verify the iso when testing. During lucid, sometimes the image did not change for 5-7 days before anyone knew it.


CategorySpec

Specs/StructuredTestingPrograms (last edited 2010-05-10 14:04:33 by 217)