UserManualTests

Objectives

Last cycle, it happened oftenly that regressions on Unity were hardware specifics, leading to even not being able to boot the session on some intel graphic hardware. All was fine on nvidia ones and nothing noticeable.

We need before releasing a new version of unity in ubuntu having tests coverage for *what the user sees*, not only relying on unity internal state (for instance, it happens that the launcher "is shown" from an unity internal state, but there is a stacking issue preventing it actually appearing on the screen).

Distro is not upstream and we want a different approach and cross check with user testing which is only the way to ensure that we checked the basic workflow of what the user is supposed to do. Those test cases have to be executed in a wide variety range of hardwares, and we can't scale ourself in that direction: involving a lot of technical users running unstable but not technical enough for making bug reports is a nice way for addressing that goal.

Consequently, we need an entertaining tool, where you can complete all of just a subset of a manual testsuite and be rewarded for it.

We need as well to analyze easily server side those data.

Idea

• Ship development version of Unity (and possibly other projects) with a set of test-cases
• Provide users of this version of Unity with an easy way to run these tests and report results
• Provide Unity and Ubuntu engineers with a way to analyze these results

Requirements

Data to gather

• version of tested software
• hardware info
  • processor type
  • gfx card
  • installed memory
  • disk type (hd or ssd)
  • resolution
  • sound card
• locale (language, ltr or rtl)
• test results
• configuration (by a runned command when test fails)

Test format

Must-have

• Ability to define requirements:
  • availability of an application
  • success of another test
  • multiple monitors
  • desktop environment (unity 2d, unity 3d, KDE...)
  • gsettings keys check for a particular configuration
• Being able to run a setup command
• Specify test steps and expected results
• Definition of an internal name (can be used by developers in commit messages)
• Different test types
  • Manual: user selects yes, no, don't understand, skip → enum
  • Automatic! command runs and prints result on stdout → bool or int (useful for benchmarks: duration, fps...)
• Group by domain (Launcher, Panel, Dash...)
• Group by pre-condition (Must be a state, not an action. Example "Desktop is empty")

Nice-to-have / ideas

• Estimated time to run a test
• Ability to comment if a manual test fails?
• Command launched to check the configuration if the test can be run or not

Possible file layouts:

• one file per domain:
  • $domain1
  • $domain2
• Break large domains in multiple files:
  • $domain1-$foo
  • $domain1-$bar
  • $domain2

Storage

• Should support Launchpad SSO to report results.
• Expose a web API to be able to build a web UI to analyze reported results.

WebUI

Must-have

• for version n: result of all tests
• evolution of one test result over time (regression spotting)
• group results by hardware (examples: "all nvidia", or "nvidia 80xx" or "arm"...)
• group results by locale

Nice-to-have / ideas

• Link to report a bug

Existing solutions

We looked into two existing solutions: Checkbox and Litmus, but did not consider them adapted to our objectives:

• Checkbox
  • (+) Collects hardware information
  • (+) Integrates with Launchpad SSO
  • (-) Monolithic: ships its own set of test-cases, not designed to run a set of tests for one project
  • (-) Complex code-base
  • (-) Not entertaining UI
• Litmus
  • (+) Tests can be provided by users

  • (+) Web-based => easy for users to get started

  • (-) Web-based => problem to gather hardware information

  • (-) Has its own user management: not integrated with Launchpad SSO

  • (-) Tests are kept outside of the code => need to synchronize versions

  • (-) User interface is tied to Firefox => would need a lot of work on the website to avoid confusing users

  • (-) Uses unfamiliar technology (Perl cgi scripts)

Implementation Notes

Using results-tracker.ubuntu.com to store results look like the best solution

API between Interface and test cases manager

• get list of domains {domains: # of tests}

• get_testruns(domain=All) => generator (testrun object, number of remaining tests)

-> next test will depend on previous tests results (fails, succeed, etc.)

• run_setup_command() -> for current testrun

testrun.failure(comment="") testrun.success() testrun.skip()