Launchpad Entry: qa-expand-checkbox-coverage
Contributors: sbeattie, apulido, schwuk, cr3
Packages affected: checkbox
Checkbox would benefit from integrating the QA regression testing suite.
This integration should make it more appealing to run the QA regression testing suite more regularly.
- Provide a larger pool of tests to be run by the security and SRU validation teams to test for general regressions when preparing updates to -security and -updates
- Create a wider base for platform team milestone testing and certification testing
Integrate test cases from other collections.
- Should other test collections be pulled in in bulk, or should we cherry-pick?
- How to organise the tests?
- How to maintain them?
BoF agenda and discussion
- in a bzr branch not a package at this point in time
- it is mostly for server packages or command line applications
- should be incorporated for testing proposed packages and development releases
- granularity - be able to run specific tests individually (a fix for a bug), then run the full test suite
Have a command line interface to checkbox to be able to run specific tests or suites of tests
Cherry picking tests is expensive because it is like forking from a project - so just start running all of the tests
- If a test fails, that failure should be recorded and future failures of that test should be expected
- create your baseline in the first run of the test suite so 45 out of 50 passed, the next test run you expect at least 45 passes - anything less than 45 is a problem
We should not write tests that have already been written, we should leverage upstream tests and trust that those tests are good
- Extending them to cover more situations is ok, though, right?