Ubuntu Wiki

• Launchpad Entry: karmic-qa-checkbox-testing-sequencing-and-management

• Created:

• Contributors:

• Packages affected:

Summary

How we can improve test sequencing - e.g. re-running tests after a suspend/resume test - and management - e.g. resuming the test sequence after a crash.

Release Note

This section should include a paragraph describing the end-user impact of this change. It is meant to be included in the release notes of the first release in which it is implemented. (Not all of these will actually be included in the release notes, at the release manager's discretion; but writing them is a useful exercise.)

It is mandatory.

Rationale

This should cover the _why_: why is this change being proposed, what justifies it, where we see this justified.

User stories

Assumptions

Design

You can have subsections that better describe specific parts of the issue.

Implementation

This section should describe a plan of action (the "how") to implement the changes discussed. Could include subsections like:

UI Changes

Should cover changes required to the UI, or specific UI that is required to implement this

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Migration

Include:

• data migration, if any
• redirects from old URLs to new ones, if any
• how users will be pointed to the new way of doing things, if necessary.

Test/Demo Plan

It's important that we are able to test new features, and demonstrate them to users. Use this section to describe a short plan that anybody can follow that demonstrates the feature is working. This can then be used during testing, and to show off after release. Please add an entry to http://testcases.qa.ubuntu.com/Coverage/NewFeatures for tracking test coverage.

This need not be added or completed until the specification is nearing beta.

Unresolved issues

This should highlight any issues that should be addressed in further specifications, and not problems with the specification itself; since any specification with problems cannot be approved.

BoF agenda and discussion

Rationale

• Need to handle crashes and recover appropriately
  • If a test fails partway through or user interrupts test, Checkbox should allow the option to resume from where it left off when restarted
    • NB: This should be an option -- users should be able to restart as well
  • If there were a crash, etc. the test that caused it should be marked as a failure so that the result can be uploaded to Launchpad

Dealing with Test Failures

• E.g. mass suspend/resume
  • If suspend/resume fails at e.g. #127, we should abort the test there rather than try to resume from #128
• Scripts should contain log information that shows how far a test got
• Ideally Checkbox should be able to tell when a machine has failed when running in an automated environment
  • This is more difficult than it sounds... how do we detect when a machine has crashed?
  • Timeout values are supported, but aren't used by default
  • Individual tests may be able to resume themselves from where they died, but that must be done by each individual test
    • No way for Checkbox to know where a test failed... it needs to log this itself

Running Tests Multiple Times

• Sometimes we want to run the same test multiple times
  • E.g. we want to run a test that does 50 pings 5 times to get 250 pings
    • Perhaps this should actually be a separate test suite so that we can see a difference between "this works for small numbers" and "this fails for large numbers"
  • Or maybe run tests continually for a certain amount of time
  • Other times we want to be able to run a script before and after some condition
    • E.g. run the network test, suspend/resume, then re-run the network test
    • In Checkbox we can define dependencies already. Perhaps we can add new keys in addition to "depends" such as "before" and "after.
      • This way lists of tests can be defined as needing to be run before or after a specific test...
        • This should be as close to real user behavior as possible
      • Where do we define this? In the tests being run, or in some other file?
        • If this is done within the test or suite definition, we end up with the problem of having circular dependencies!
• It might also be useful to have the concept of threaded tests
  • E.g. we want to have music playing while running the suspend test, then when we resume check to see that the music is still playing
  • This might not be a Checkbox concern so much as a test suite concern -- test suites might want to amp off a bunch of apps and check on them after some action is taken
  • There are OSS packages, "Staf" and "Stax" that allow definition in XML of tests to be run in serial and parallel

    • http://staf.sourceforge.net

Stress Tests

• Randomization of tests
  • E.g. "Run tests randomly until I tell you to stop"
  • Allows good long-term stress-testing
  • Would require good logs showing order of tests executed and under what conditions they failed and passed
  • Could we have a script or something that generates a test suite of x number of tests and uses the serialization of tests functionality (described above) to execute these? In that case we get the random effect but don't have to add that level of complexity into Checkbox

Showing Progress

• Would be nice to show progress bar or something as tests are run
  • Otherwise users might think that a long-running test has hung
• UI: should have a disclosure triangle, similar to the "Details" one on Synaptic/gdebi
  • Dave Murphy suggests talking to the DUX team about this
• Should show amount of time left or percentage of test(s) performed in the progress bar
  • This might not be a nice user experience as we might skip tests if tests they depend on fail
    • jumpy progress bar
  • Maybe "tests remaining"?
  • Sounds like another good question for the DUX team
  • Implementation idea: an optional plugin that is capable of assessing a running test and reporting back that information to Checkbox

CategorySpec