To be able to improve the quality of our automation strategy we need to know how many bugs are found during test automation activities and how many during other types of testing. This applies for now, only to the canonical-platform-qa team, the community can continue to raise bugs normally.

These tags are meant to help analyze the effectiveness of the automated testing that is being put in place, as well as help developers distinguish the defects that have been found by known test cases.

Types of Automated Testing

For all the testing types that we do for Ubuntu, the aim is to automate everything that makes sense and will make testing more efficient.

The three main types of testing we do are:

• Image testing: testing done on images, regardless of whether it is manual or automated.

• SRU testing: testing done on SRUs, regardless of whether it is manual or automated.

• Upgrade testing: testing done on upgrades, regardless of whether it is manual or automated.

The different automated test suites that we will be generating/using are:

• Daily image testing: smoke tests that run to make sure the basic functionality is there and determine whether it is worth running further testing. Installation of the system is part of the daily image testing.

• QA Full Test Suite: this will be the collection of test cases that will form our regression test suite. They do not exist as such yet, but we'll start adding test cases to this collection as soon as smoke testing is in place.

• Manual testing: Whenever we do exploratory testing or any kind of manual testing activity, we'll mark the bugs as such. This tag must not be used for manual verification of a problem found with automated testing.

Tagging Bugs

How to tag bugs found during our testing activities, for test escape analysis and quality purposes, as well as for developers knowing which kind of testing found the problem.

Always tag the bugs with the affected release code, but if a bug is found in the latest release and is known to be also affecting previous ones, we should tag the bug for the previous releases too too.

Ad-hoc and exploratory testing will be marked as qa-manual-testing, and any other tag that applies.

For clarity:

• qa-daily-testing + iso-testing = ISO daily smoke testing
• qa-daily-testing + dist-upgrade = daily upgrade testing
• qa-daily-testing + qa-sru-testing = daily regression testing

Criticality of problems

Depending on whether the bug breaks the build or not, the criticality will be decided.

For a build that is unusable for testing/development usage, the bug will be "Critical".

As a guidance, a qa-daily-testing bug will be almost certainly "Critical" or "High".

For any daily testing problem, the QA engineer needs to subscribe to the bug and make sure the right people to deal with the bug get notified. We need to be pushing for these defects to get the right attention sooner rather than later. Our ISO watching job includes the getting the defect fixed asap follow up.