Server

Differences between revisions 6 and 7
Revision 6 as of 2016-12-08 12:59:16
Size: 5942
Editor: powersj
Comment: Completed bug section
Revision 7 as of 2016-12-08 13:11:32
Size: 5938
Editor: powersj
Comment:
Deletions are marked like this. Additions are marked like this.
Line 18: Line 18:
This testing occurs on Jenkins instance using jobs built by [[https://code.launchpad.net/~canonical-platform-qa-jenkins/qa-jenkins-jobs/trunk|Jenkins Job Builder files]] and here is the code used to validate the [[https://code.launchpad.net/~ubuntu-server-qa/ubuntu-test-cases/server-tests-raring|preseed and tasksel]]. This testing occurs on Jenkins instance [[https://code.launchpad.net/~canonical-platform-qa-jenkins/qa-jenkins-jobs/trunk|using jobs created by Jenkins Job Builder]] and the [[https://code.launchpad.net/~ubuntu-server-qa/ubuntu-test-cases/server-tests-raring|test cases]] used to verify the preseed and tasksel.
Line 23: Line 23:
Testing is outlined by point-release milestone. Inside each milestone has a list of the architectures to test and inside there a list of test cases. Above each test case there is a link to the specific ISO that needs to be tested. The layout looks like the following: Testing is outlined by point-release milestone. Inside each milestone has a list of the architectures to test and inside there a list of [[https://bazaar.launchpad.net/~ubuntu-testcase/ubuntu-manual-tests/trunk/files/head:/testcases/image/|test case]]. Above each test case there is a link to the specific ISO that needs to be tested. The layout looks like the following:
Line 28: Line 28:
     |
Line 30: Line 29:
          |
Line 38: Line 36:
          |
Line 44: Line 43:

The [[https://bazaar.launchpad.net/~ubuntu-testcase/ubuntu-manual-tests/trunk/files/head:/testcases/image/|test cases]] are available for updating.
Line 76: Line 73:
 * Write dep8 tests for packages you use or have installed
 * Triage bugs
 * Work on backlog bugs
 * Write dep8 tests for packages you use, have installed, or to help out
 * Triage bugs for enough information and actual issues
 * Work on the backlog bugs

Overview

Areas under test and opportunities for testing of Ubuntu Server. At the end of this document is a list of opportunities.

ISOs and Cloud Images

As the delivery vehicle for obtaining Ubuntu Server the ISOs and cloud images must be tested. This is the first touch users have with Ubuntu Server and therefore a positive user experience is essential. ISOs are produced on two release cadences:

  • Daily ISOs are produced to test the latest releases

  • Point Release ISOs are produced to make updates to install components to fix bugs or add hardware support

Automated Testing

Testing of the daily ISOs is done in an automated fashion whenever a new "pending" daily ISO is produced. The testing is done across both the amd64 and i386 architectures, although work is underway to do the same for ppc64el. This automated testing consists of the following:

  1. Download and verify hashes
  2. Smoke test of a single, default install to verify no obvious issues
  3. Promote "pending" to "current"
  4. Installs with a variety of preseed and tasksel options set to validate specific software packages and configuration settings.

This testing occurs on Jenkins instance using jobs created by Jenkins Job Builder and the test cases used to verify the preseed and tasksel.

Manual Testing

Because of the importance of the ISO experience manual testing is still required. This testing occurs via the ISO Testing Tracker.

Testing is outlined by point-release milestone. Inside each milestone has a list of the architectures to test and inside there a list of test case. Above each test case there is a link to the specific ISO that needs to be tested. The layout looks like the following:

    Xenial 16.04.1
     |
     |--> amd64
          |
          |--> Install (Default)
          |--> Install (Default + crypted LVM)
          |--> Install (Default + LVM)
          |--> Install (Bind9)
          |--> Install (Default + RAID1)
          |--> etc.
          |
     |--> i386
     |--> ppc64el
     |--> powerpc
     |--> arm64
     |--> s390x

Packages

The server team's list of owned packages need testing. The first and foremost option is to create and utilize dep8 tests. Writing these tests help verify standard functionality and that the packages are usable. Many packages do not have tests written at all and some could use even more tests.

Bug Work

As a part of owning quality it is important to stay involved in reported defects. During the defect resolution process, bugs should have tests verifying that the bug is actually solved. These tests should then be put back into the package's tests (e.g. dep8) or pushed back to the upstream package's project.

Daily Triage

There is an effort by the Server Team to deal with server specific bugs.

Backlog Triage

Triage consists of a rotating team member who is in charge of sorting through the list of new bugs. This consists of checking that enough information to go work on the bug and that it is a legitimate and valid problem in Ubuntu Server to solve. If there is not enough information or the issue is not a bug, they work with the reporter to either gather additional information or close the bug.

Backlog Work

Once a bug is validated and known to be reproducible it is added to the backlog of bugs. These bugs will have at the very least steps to reproduce and are ready of work. They also may be very simple as they already have a patch that needs to be accepted. In either case, these bugs are ready for work.

Usage

Above everything, one of the best ways to work on quality in Ubuntu Server is to use it!

Personal Scenarios

Deployments

Live Patch

Opportunities

If you have questions about any of these, please ask in the #ubuntu-server channel on freenode.

  • Automated ISO testing can always take on more test cases when appropriate
  • Manual ISO testing is always needed. Sign-up on the QA tracker, pick a release, and complete test cases. There is a calendar on the left that can help coordinate when a new release is coming up and testing will really be needed.
  • Write dep8 tests for packages you use, have installed, or to help out
  • Triage bugs for enough information and actual issues
  • Work on the backlog bugs

Testing/Server (last edited 2017-01-26 18:23:44 by powersj)