Ubuntu Wiki

• Launchpad Entry: karmic-qa-test-tracker and karmic-qa-checkbox-testcase-integration

• Created:

• Contributors:

• Packages affected:

Summary

We will work with ISD to develop a system used for organising test cases and tracking a wide range of test results. The development is driven by ISD with Platform QA as a key setter of requirements.

Release Note

Improved test case and result tracking are being developed to improve the quality of Ubuntu.

Rationale

The various testing and QA efforts around Ubuntu and Launchpad have a number of test case formats and are tracking their results in different places. This makes it difficult to get an overview of all of the testing and results that is going on, leading both to duplicated effort and missed testing opportunities. Uniting the test cases and results in one central repository will help promote better testing methods and foster communication between teams.

User stories

• Members of the QA team are performing a number of tests that can be useful to other teams. They would like to make sure that these results are published in a way that everyone interested can see them.
• Members of the OEM team are conducting tests that frequently overlap with tests the QA team are conducting. They would like to share efforts and be certain that all test cases are properly written and cover the test conditions properly.
• Members of the community have unique insights into issues that are not encountered within internal Canonical teams. They would like to add or improve test cases to extend coverage.

Assumptions

Design

The notes from UDS (below) will be used as a starting point to discuss the design of the Test Tracker and of the Test Case repository.

BoF agenda and discussion

Use Cases

• a number of teams need to track test results
  • ISO tests
  • package tests
  • various QA tests
  • etc.
• quickly gauge result metrics between multiple testers
• simplifiy triaging tests between QA members
• measure workload of team members
• discover which automation is most fragile and breaks down most frequently
  • useful for making more robust automation
• highlight which features cause the most defects between releases
• history of results allows metrics that quantify quality
• can also show up-to-date snapshot of quality status

Goals for Discussion

• determine needs for test tracker that satisfies all various use cases
• plan (or start to plan) how to implement this

Requirements

• test case management
  • manual and automated
• results management
  • receiving raw results from test systems or users
  • triaging failures

Current Tools

• Launchpad team
  • wiki
    • teams have various formats, not consistent across group
    • a lot of ad-hoc testing, little formality of test plans
  • bugs
• OEM team
  • wiki for test cases
  • Checkbox for tests
    • want to move to integrate with certification server to track results
    • currently tracking via webpages
  • pain points:
    • producing wiki tables is painful due to formatting issues
      • so painful that they have considered writing a script to parse the wiki and detect syntax errors
      • also considered greasemonkey scripts to generate wiki tables
• QA ISO testing

  • using web app - http://iso.qa.ubuntu.com/

    • administrators update image information and test cases
    • e-mail notification of new builds
    • test cases link to wiki page
    • users run test cases and can link failures to Launchpad bugs
    • overview page to get status; detail pages by milestone and image
    • pain points
      • Launchpad integration is flaky (doesn't use launchpadlib)
      • no way to cancel a mis-filed report (no delete or move functionality)
      • no good way to coordinate testing
        • no way to hold testers accountable for tests
        • no way to mark a test as in-process, leading to duplicate effort
      • overlapping test cases
• Checkbox / certification system
  • provides a harness for automation of tests
  • allows results to be collected from automated and "facilitated manual" tests
  • results reported to a central database
    • allows querying by machine, image, etc. to show results

Solution Ideas

• System that interoperates with wiki but adds structure
  • wiki page contains test case
  • click a button on a webpage, that submits it to a tracker to become a testable case
  • tests read in by a system that can convert it to a Checkbox (manual) test
    • can then run test manually by reading wiki page
    • or can run through checkbox
    • or can run with a printout of the testcase and report results later
  • can do test version tracking within wiki (we basically get this for free)
  • pro: we already have a lot of expertise in working with the wiki structure and parsing data from it
  • need a way to ensure that test cases are in the correct format
    • have an extra-wiki interface that formats the test case correctly and puts it in the wiki
  • we will also need to be able to take all the existing cases and get them into this central system/format
• Wiki-based "database" with Checkbox frontend
  • can work off of above system -- if we have a consistent format in the wiki that can be parsed, Checkbox can import data from it

  • should not be the only method of getting data in, as this won't work for all cases (e.g. ISO testing)

  • from website can select tests and export as a Checkbox suite, import into Checkbox and run
  • Checkbox would need the ability to report results to the tracker website (or database or wherever they get stored)...
• Need standard format for test cases before we can implement any ideas
  • common set of key/value pairs among teams
  • "What does a test case contain?"
    • title
    • summary
    • tests....
  • process discussions of how to write a test case?

Questions

• How do you connect test cases to test results?
  • Need to have some sort of unique identifier for each test case
• Where do the results get stored?
  • Wherever it is, it needs to be linked to the test case descriptions in the wiki

Discussion Note

This is discussion continued into another UDS session; notes from this discussion are below.

Specifications and Test Cases

• Would it be useful to have a spec for Ubuntu as a whole? (At a high level)
  • We could start this from existing test cases and grow from there
  • should be reviewed every release or when needed
  • could be based for each "general class" of release
    • UNR, desktop, server
  • is it useful to look at this for Ubuntu as a whole? It might be more useful to look at this on a package level
    • Ubuntu is more a collection of specs and changes; we're not always in control of the direction of everything
• Does .deb format accept/support enough metadata to link to test cases?
  • .deb could contain information / links to test cases
• How can we derive meaningful information from specifications?
  • Have the concept of overall goal and then specific goals for specific releases?
  • "Rebasing" of specs -- could have previous spec and a diff, then after release the spec + diff becomes the new baseline (and is then diff'ed again)
• Consider allowing external agencies to give weight to specs or test cases
  • e.g. something reported by someone with 0 karma might be ranked lower than something that's been reopened
• Consider leveraging all test cases in wiki. Generate a random selection of tests and distribute them (through Checkbox?) to community members
  • Tests that are consistently skipped are "bad" tests and should eventually be removed
  • Consensus is against this -- there are not enough data provided about why tests are being skipped; a human needs to make this decision, not a script
• Test weighting can be useful
  • Certain test need to be performed always and/or multiple times, others might need to be performed only occasionally
  • Test cases can be generated/distributed taking this into account so not everything gets repeated
• Test cases should be atomic
  • individual IDs
  • test plans can be determined based on need using atomic test cases
    • different groups might weight tests differently and require or skip different tests

Requirements

• preferably written in Python
• open-source
• ability to create bug in Launchpad from a specific failure
• ability to link a spec to test cases?
  • specs might change or not be specific enough to create meaningful test cases
  • we won't have specs for everything that comes in, and demanding this will annoy developers and not give great results
• need to be able to triage tests
  • expire/remove obsolete tests
  • possibly rank "good"/"bad" test cases

Existing Test Result Tracking Systems

• testlink
  • PHP, open-source, DB backend
  • takes test cases from requirements
    • input requirements, link test cases to requirements, link test cases to builds
  • cons:
    • written in PHP
    • no triaging of test failures
    • no guided testing
    • does not support attachments everywhere
• racetrack
  • allows triaging of test failures
    • e.g. test script failure, re-ran and it worked
  • supports arbitrary attachments
  • cons:
    • no guided testing
    • no ACLs
• launchpad
  • Chris Gregan has considered seeing how difficult it would be to strip down launchpad to use as result tracker
  • could reuse some launchpad features

  • would natively support launchpad integration

  • cons:
    • would be very difficult to implement
• certification website
  • written in Python
  • integrates with launchpad
  • developer has same needs as requirements listed
  • cons:
    • would require some development (not immediately ready to go)

Action Items

• Discuss on QA CoP
  • Determine an open channel for discussion
  • Agree on requirements for test cases
    • this will lead to standard test case format across teams

CategorySpec