TestcaseWikiMigration

Differences between revisions 1 and 11 (spanning 10 versions)
⇤ ← Revision 1 as of 2008-12-15 13:03:36 →
Size: 3618
Editor: cpc1-oxfd8-0-0-cust993
Comment: new spec
← Revision 11 as of 2009-01-13 10:54:46 → ⇥
Size: 4118
Editor: cpc4-oxfd8-0-0-cust39
Comment:
Deletions are marked like this. Additions are marked like this.
Line 12: Line 12:
This should cover the _why_: why is this change being proposed, what justifies it, where we see this justified. We need a consistent syntax to define the test cases and a better way to track them. Right now, test cases are maintained under https://wiki.ubuntu.com/Testing/Cases but the syntax is not defined and there is no logical way to track what it is available.
Line 16: Line 16:
== Assumptions ==  * Mark, a netbook-remix user, wants to add a new test case for an application that already has a test case for Ubuntu desktop. He calls the already available test case from the UNR page, needing only to upload the netbook-remix specific screenshots.
 * Anne, a Kubuntu user, finds a bug in Kpdf while reading a complex PDF file. She creates a LP bug but also creates a new test case to track future regressions. This file will be added also as test case for Evince in Ubuntu.
 * Chris wishes to track the testing for a mobile build. He can do that in the wiki by using a macro
Line 20: Line 22:
You can have subsections that better describe specific parts of the issue. === Testcases Syntax ===

A standard test case template will be drawn up which will be used by all test cases. [[/ApplicationTemplate|Application Template]]
The syntax will not be automatically checked, at this point it is enough to follow the template and reedit them manually if errors are found.

==== Metadata ====

  * Distro(s) - for which distro flavours is the case valid?
  * Version(s) - for which distro versions is the case valid?
  * Application name
  * Attachments required for the test case should be stored in the wiki and linked explicitly

=== Testcase Number ===

After much discussion it has been agreed that we will use the following syntax to track the test cases number as:
 * If it is a test case for an application, then the testcase number will be [a-z]{2}-[0-9]{3} (2 letters for the application and 3 numbers for the test case number)
 * If it is a test case for a Launchpad bug then the testcase number will be lp-[bugnumber]

The testcase number will be checked for syntax and a warning will be raised in these cases:
 * No track number has been added
 * The track number is already used by other testcase.

=== Path Structure ===

We have decided to move the pages from {{{/Distribution/Applications}}} to just {{{/Applications}}}. For example, the Ekiga page will be at {{{http://testcases.qa.ubuntu.com/Applications/Ekiga}}} instead of {{{http://testcases.qa.ubuntu.com/Ubuntu/Applications/Ekiga}}}.

A Moin Moin plugin will be needed to change to the proper screenshots when called with /Ubuntu, /Kubuntu, /Xubuntu, etc.

The testcases from pulled from launchpad bugs exists under http://testcases.qa.ubuntu.com/Regressions/ with per-data type subtrees, but is open to renaming or restructuring.

=== Tracking results ===

A test tracking facility in a separate page tree but in the same wiki will make it easy to track test results linked to the cases.

 * Use a standard page space such as {{{/TestRuns/<identifier>}}} to collect results
 * Write a macro to list the tests used (and optionally display the full case text) and collect the results through simple input boxes
Line 24: Line 61:
This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:  '''Admin:''' A new LP project needs to be created to track bugs and code for the custom changes (plugins, themes and macros).
Line 26: Line 63:
=== UI Changes ===

Should cover changes required to the UI, or specific UI that is required to implement this

=== Code Changes ===

Code changes should include an overview of what needs to change, and in some cases even the specific details.		  * '''Displaying case numbers''' - A ''Macro'' for collecting and displaying case numbers and titles in table form
 * '''Test results''' - A '''Macro''' based on [[http://moinmo.in/MacroMarket/PageComment2|PageComment2]] to display case information with result entry boxes and result tables. The actual results are stored in a sub-page (as the comment macro does).
Line 36: Line 68:
Include:
 * data migration, if any
 * redirects from old URLs to new ones, if any
 * how users will be pointed to the new way of doing things, if necessary.

== Test/Demo Plan ==

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

== Unresolved issues ==

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.		  * After migration all the old test cases wiki pages need to have just an advice pointing to the new test cases wiki.
Line 52: Line 71:
Testcases in wiki.ubuntu.com: https://wiki.ubuntu.com/Testing/Cases

New testcases specific wiki at http://testcases.qa.ubuntu.com

 * Syntax of testcases
 - Way to add attached documents or additional files to the test case.
 - https://testcases.qa.ubuntu/Distro/Applications/Application seems a natural way to order the wiki
 - We could have a macro that pulls the correct screenshots for a test cases depending on the distributuion:
  i.e. We cave testcases.qa.ubuntu.com/Applications/Gedit, but when called from /Xubuntu/Applications/Gedit it will contain Xubuntu screenshots if available.
 
 * Syntax of the track number
 - Right now is going u-ff-001 (u for ubuntu, ff for the app (in this case firefox), and 3 number for the test case numbers)
 - Change that to ff-001 (2 letters for the application and 3 numbers for the test case number)

 * How to use macros to encourage people to contribute in a easier

OEM/UNR Testcase format example:
https://wiki.ubuntu.com/Testing/Cases/Ubuntu-Netbook-Remix

How to track test cases run?
 - Chris maintains a table with results and then use a python script to parse it and convert it if necessary.
 - [Chris] We need to use testcases.qa.ubuntu.com to track results of tests. Maybe with macros. (in a ISO tracker manner)
 - ISO tracker and testcases.qa.ubuntu.com communication
 
 
Are testcases wiki tasks handled in LP? Do we have a project to file bugs?
 - Dave to set up a LP project to file bugs and have the code in bzr
 
 http://moinmo.in/

Meta Data for case suite creation - Meta codes for each case:
 - Distro(s)
 - Version(s)
 

Code format - ff-001 = Ubuntu Jaunty, Firefox, Case 1		 ''merged into spec''

  • Launchpad Entry: qa-testscase-wiki

  • Created: 2008-12-15

  • Contributors: heno, schwuk, apulido, sbeattie

  • Packages affected:

Summary

Migrating all test cases to testcases.qa.ubuntu.com, adding useful macros and making sure all relevant teams are using the new facility.

Rationale

We need a consistent syntax to define the test cases and a better way to track them. Right now, test cases are maintained under https://wiki.ubuntu.com/Testing/Cases but the syntax is not defined and there is no logical way to track what it is available.

Use Cases

  • Mark, a netbook-remix user, wants to add a new test case for an application that already has a test case for Ubuntu desktop. He calls the already available test case from the UNR page, needing only to upload the netbook-remix specific screenshots.
  • Anne, a Kubuntu user, finds a bug in Kpdf while reading a complex PDF file. She creates a LP bug but also creates a new test case to track future regressions. This file will be added also as test case for Evince in Ubuntu.
  • Chris wishes to track the testing for a mobile build. He can do that in the wiki by using a macro

Design

Testcases Syntax

A standard test case template will be drawn up which will be used by all test cases. Application Template The syntax will not be automatically checked, at this point it is enough to follow the template and reedit them manually if errors are found.

Metadata

  • Distro(s) - for which distro flavours is the case valid?
  • Version(s) - for which distro versions is the case valid?
  • Application name
  • Attachments required for the test case should be stored in the wiki and linked explicitly

Testcase Number

After much discussion it has been agreed that we will use the following syntax to track the test cases number as:

  • If it is a test case for an application, then the testcase number will be [a-z]{2}-[0-9]{3} (2 letters for the application and 3 numbers for the test case number)
  • If it is a test case for a Launchpad bug then the testcase number will be lp-[bugnumber]

The testcase number will be checked for syntax and a warning will be raised in these cases:

  • No track number has been added
  • The track number is already used by other testcase.

Path Structure

We have decided to move the pages from /Distribution/Applications to just /Applications. For example, the Ekiga page will be at http://testcases.qa.ubuntu.com/Applications/Ekiga instead of http://testcases.qa.ubuntu.com/Ubuntu/Applications/Ekiga.

A Moin Moin plugin will be needed to change to the proper screenshots when called with /Ubuntu, /Kubuntu, /Xubuntu, etc.

The testcases from pulled from launchpad bugs exists under http://testcases.qa.ubuntu.com/Regressions/ with per-data type subtrees, but is open to renaming or restructuring.

Tracking results

A test tracking facility in a separate page tree but in the same wiki will make it easy to track test results linked to the cases.

  • Use a standard page space such as /TestRuns/<identifier> to collect results

  • Write a macro to list the tests used (and optionally display the full case text) and collect the results through simple input boxes

Implementation

  • Admin: A new LP project needs to be created to track bugs and code for the custom changes (plugins, themes and macros).

  • Displaying case numbers - A Macro for collecting and displaying case numbers and titles in table form

  • Test results - A Macro based on PageComment2 to display case information with result entry boxes and result tables. The actual results are stored in a sub-page (as the comment macro does).

Migration

  • After migration all the old test cases wiki pages need to have just an advice pointing to the new test cases wiki.

BoF agenda and discussion

merged into spec

CategorySpec

QATeam/Specs/TestcaseWikiMigration (last edited 2009-01-13 10:54:46 by cpc4-oxfd8-0-0-cust39)