KernelAutomatedTests

Summary

There are existing kernel test scripts which should be integrated into checkbox and run as a daily smoke tests and monthly performance/stress tests in the certification lab. We need to identify these tests and outline the necessary steps to get this running on in an automated fashion.

Release Note

In an effort to provide a high quality LTS release for Lucid 10.04, the QA Team has incorporated a series of daily kernel smoke tests and monthly performance/stress tests into checkbox. These have began running on machines in the certification lab and results are being propagated to the Ubuntu Kernel Team.

Rationale

With Lucid 10.04 being an LTS release there is a greater focus for producing a high quality and stable release. As a result, it only makes sense to provide more testing as early and as often as possible. It's been identified that there are existing kernel tests which are not currently being ran against systems in the certification lab. These tests should be identified and integrated into checkbox and subsequently ran on a consistent basis through Lucid's development cycle.

User stories

  • Joe had decided to upgrade to Lucid Beta 1 and notices his system fails to suspend/resume properly. It happens that his same system also exists within the certification lab. If a suspend/resume test script had been integrated into checkbox and ran on a consistent basis leading up to Beta 1, this issue would have already been flagged and hopefully addressed.
  • Sue is somewhat of an early adopter and decides to upgrade to Lucid Alpha 2. Sue notices one of the kernel security regression tests is failing. This is never good. Had the kernel security regression tests been integrated into checkbox and running, this regression would have already been caught and resolved much sooner.

Design

  1. Identify kernel tests to be integrated into checkbox
  2. Wrap each kernel test into a checkbox test case/suite
  3. Integrate/Merge kernel tests into checkbox
  4. Begin running daily kernel tests in an automated fashion on systems in the lab
  5. Report results to the Ubuntu Kernel Team

Implementation

  1. Identify kernel tests to be integrated into checkbox
    • security and kernel regression tests - bzr branch lp:qa-regression-testing (test-kernel.py test-kernel-security.py)

    • wakealarm test script - available in cking git tree - http://kernel.ubuntu.com/git/cking/scripts.git

    • suspend/resume test script - already in checkbox just needs a test case
    • network test script(s) - already in checkbox just needs a test case
    • LTP - already in checkbox but currently disabled
      • [cr3 ]Fixing the LTP build
      • [cr3] Creating a blacklist of LTP tests
    • Autotest - already in checkbox but not enabled
      • [leannogasawara] Create a list of tests to run from autotest
  2. Wrap each kernel test into a checkbox test case/suite
  3. Integrate/Merge kernel tests into checkbox
  4. Begin running daily kernel tests in an automated fashion on systems in the lab
    • [cr3] Initial daily kernel tests running by Lucid Alpha1
    • [cr3] Evaluate EC2/UEC integration
    • [leannogasawara] gcov kernel testing to evaluate kernel coverage (ogasawara to build kernel, talk to fader for testing)
  5. Report results to the Ubuntu Kernel Team
    • [needs owner] Test results (ie failures) propagated back to the Ubuntu Kernel Team (maybe fader? can help)

Code Changes

  • [cr3] Problem: the kernel tests (most of them) need sudo
    • workaround: in certification labs, we know the user/password and can hard-code it. we still need a good way to do this so the community can run these tests with proper credentials prompting
  • [cr3] Checkbox does not support yet the daily v.s. monthly idea (planned for Lucid)
    • Once this is available we can then leverage this for daily smoke testing as well as monthly performance, stress, code coverage testing

Test/Demo Plan

These will initially only be run on systems within the certification lab. In the future, users should eventually be able to run these tests as any other regular checkbox test (System->Administration-System Testing).

Unresolved issues

None.

BoF agenda and discussion

Objective: integrate kernel team tests into checkbox

Leann has been working on the scripts They should be integrate in a test suite

  • kernel smoketests (run daily, as they go quickly)
  • stress tests (run less frequently, more involved)

Problem: the kernel tests (most of them) need sudo

  • workaround: in certification labs, we know the user/password and can hard-code it
    • we still need a good way to do this so the community can run these tests with proper credentials prompting

Tests in version control:

  • Create a test suite
  • Get the tree

The kernel team has around 4-5 testsuites wakealarm tests suspend/resume tests

Checkbox does not the concept of diverging from a previous snapshot Checkbox does not support yet the daily v.s. monthly idea (planned for Lucid)

NB: UEC and EC2 kernels are different -- testing UEC does not necessarily help EC2

LTP Autotest - it is available, but not enabled (it takes a long time to run)

ACTIONS

* Creating a blacklist of LTP tests * Fixing the LTP build * Evaluate EC2/UEC integration (cr3) * gcov kernel testing to evaluate kernel coverage (ogasawara to build kernel, talk to fader for testing) * Create a list of list of tests to run from autotest (ogasawara, cr3) * Clean up and commit kernel team scripts (ogasaware, cr3)

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.

CategorySpec

Specs/KernelAutomatedTests (last edited 2009-12-02 20:36:50 by c-76-105-148-120)