• Launchpad Entry: qa-expand-checkbox-coverage

  • Created: 2008-12-05

  • Contributors: sbeattie, apulido, schwuk, cr3

  • Packages affected: checkbox


Checkbox would benefit from integrating the QA regression testing suite.


This integration should make it more appealing to run the QA regression testing suite more regularly.

Use Cases

  • Provide a larger pool of tests to be run by the security and SRU validation teams to test for general regressions when preparing updates to -security and -updates
  • Create a wider base for platform team milestone testing and certification testing



Integrate test cases from other collections.


  • Should other test collections be pulled in in bulk, or should we cherry-pick?
  • How to organise the tests?
  • How to maintain them?


Candidate suites:

  • Security team collection
  • Permission checker (to avoid #288479 and #290798)

BoF agenda and discussion


  • in a bzr branch not a package at this point in time
  • it is mostly for server packages or command line applications
  • should be incorporated for testing proposed packages and development releases


  • granularity - be able to run specific tests individually (a fix for a bug), then run the full test suite

Have a command line interface to checkbox to be able to run specific tests or suites of tests

Cherry picking tests is expensive because it is like forking from a project - so just start running all of the tests

  • If a test fails, that failure should be recorded and future failures of that test should be expected
  • create your baseline in the first run of the test suite so 45 out of 50 passed, the next test run you expect at least 45 passes - anything less than 45 is a problem

We should not write tests that have already been written, we should leverage upstream tests and trust that those tests are good

  • Extending them to cover more situations is ok, though, right?


QATeam/Specs/CheckboxIntegrateRegressionTesting (last edited 2009-11-10 15:19:11 by pool-71-114-235-234)