UnstableTests

Process to promote an image with failing tests

The purpose of this process is to be able to promote an image with failing tests, when the test has been identified as a non-regression, and fix these unstable tests to make them reliable.

  1. Get a list of failures from the CI dashboard
  2. For each failure on the dashboard
    1. CI file a bug against the project with the failure
    2. CI either run the test manually or ask dev (test must be run by hand) 1

    3. QA documents manual steps if not already documented. 2

  3. If --
    1. test passes in manual execution:
      1. CI retargets the bug against the infrastructure 3 [QA or CI owns / investigates]

      2. Promote the build if there is no other regression
    2. test fails in manual execution:
      1. CI retargets the bug against the production code/application [Dev owns / looks]
      2. Reject the build

Questions

  • Q: What is the incentive for people to fix broken tests?
  • A: Give people 2 weeks to fix the test, after this time, block the build

Landing email changes

  • Include bug number for every test with problems
  • date of each bug
  • time it has been open
  • before an issue is removed from the landing email, QA verifies how this has been fixed.
  1. manually: In this context manually means running the test step by step on a device without autopilot or any form of automation. (1)

  2. QA will document the tests for the failures that occurred during the last 7 days as one off to establish the base line. (2)

  3. infrastructure: CI and Autopilot (3)

CI/UnstableTests (last edited 2014-06-05 10:00:43 by brendan-donegan)