The test script infrastructure clearly has its limitations, primarily in being fragile to changes in the application being tested and run-time conditions. We must there fore take a pragmatic approach in working with the available tools but focus on the goal of testing and improving the core Ubuntu applications.

Script recording

The script recording tools are good for quickly generating the bulk of a script or recording smaller script fragments, but the resulting code will often need adjustments to run at all.

Adding robustness


Working with scripting problems

Accessibility bugs

We are currently using the dogtail test framework, which leverages the accessibility infrastructure in GTK - ATK, as do the main alternative frameworks - LDTP, Strongwind and Macaroon. This infrastructure was designed not primarily for testing but to provide improved access to the desktop for users with a range of disabilities. ATK serves this purpose quite well, though there are performance issues, bugs and incomplete support in applications. ATK has in turn been leveraged by the testing community to provide a way to drive applications and read their state. In some cases this does not perform as well as one might expect, but it is important to keep in mind that this is not the primary design goal of ATK.

Incomplete ATK support in applications also makes testing more difficult since dogtail must rely more on run-time heuristics which are in turn more fragile. Deficiencies in accessibility metadata should be reported as bugs against the target applications.

Test framework limitations

The test framework like dogtail also have inherent limitations, like not supporting drag-and-drop. These are best dealt with using work arounds including adding manual intervention steps.

Application bugs

Report these and cite the bug number as a comment in the script.

Work around comments

When implementing a work around (or hack) it's important to comment it clearly as such so the weakness can be recognized by future contributors.

{{{### FRAGILE ### The following code is fragile making the script less robust than it could be }}}

{{{### MANUAL INTERVENTION REQUIRED ### Requires user input }}}

{{{### APPLICATION BUG ### <LP bug number> }}}

File and directory structure

  • Each application to be tested has its own directory containing the following files:
    • script-list.txt - listing scripts with notes

    • The scripts themselves as <application-name-test-name>.py

    • A data directory (optional)

    • An output directory (optional)

Useful scripting tricks

Control statements

  • run(application) - a clean way to start an application

  • time.sleep(2) - pauses execution for 2 seconds

Open Questions

  • What's the best way to give an application focus? (real wm focus, not just internal dogtail focus)
  • Is there a robust way to open a file from a known path into a GUI app like OpenOffice?

Testing/Automation/ScriptingGuide (last edited 2008-12-23 00:11:54 by rhass)