DesktopAutomatedTests

Differences between revisions 7 and 8
Revision 7 as of 2007-11-15 23:49:00
Size: 5497
Editor: a91-154-119-10
Comment:
Revision 8 as of 2007-11-15 23:56:05
Size: 5843
Editor: a91-154-119-10
Comment:
Deletions are marked like this. Additions are marked like this.
Line 92: Line 92:
 * Pick one approach and implement the desktop tests in ["Testing/Cases"] with it. Skip the tests that require network access.  * Pick one approach and implement the desktop tests in ["Testing/Cases"] with it. Skip the tests that require network access. Implement a "test runner" which can execute each test script and verify that the result is acceptable. For the ATK-based tools, the scripts would be in Python, and success would be measured by a) the script finishing without errors and b) optionally running a verification function (to check that a deleted file ends up in .Trash, for example).

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

Summary

Leverage the ATK framework to implement a set of automated test routines for the desktop.

Release Note

This release contains the beginnings of an automated desktop test suite for GNOME/GTK software that support accessibility. The test suite can be used by users as well as developers to verify that the installed system works.

Rationale

At the moment, pretty much all quality assurance related testing in Ubuntu is done manually. There are wiki pages with checklists of steps that testers follow to verify that at least basic functionality works. This works, but requires a lot of people to do the work, and it needs to be done many times. Obviously, an automated solution would be better.

Testing all of Ubuntu involves many things. This blueprint is about testing desktop functionality of an installed system, including the live CD. It does not cover testing of the installer.

Use Cases

  • Aaron wants to upload a new version of the libgnomepanzerwagen package (a library that is used by many GUI programs), and wants to verify before the upload that it doesn't break anything. He installs the new version on his development machine, and runs the automated desktop tests for the entire distribution to see that everything still works.
  • Bertha works on Ubuntu QA and wants to check that the systems installed by daily ISO builds work. She installs each of them in a virtual machine, and runs the desktop test suite to verify the results.
  • Charlie supplies her customers with working systems based on Ubuntu, but also using some additional software she supplies herself. A new version of Ubuntu has been released, and she wants to make sure her customers can upgrade safely. She runs the upgrade on a test system, configured identically to those of her customers, and uses the test suite before and after to verify that nothing breaks. In addition, she has added some tests of her own, to verify that her own software also survives the upgrade.
  • Diogenes wants to help Ubuntu development and reports a bug he has found. To show how the bug can be reproduced, he starts a recording tool, does whatever is necessary to reproduce the bug, and the attaches the recording to the bug report. Developers can then use this to verify that the bug exists, and that their fix actually fixes the bug.

Assumptions

The test suite is run as a non-priviledged user on an installed system. The live CD is one such environment, but any installed Ubuntu (GNOME) system should suffice. The system may run on real hardware or in an emulator.

Design

The initial approach is to use the accessibility layer of the GTK/GNOME library stack (the ATK layer) to implement this. ATK allows programs to operate other programs more or less as if the user was operating them. There are several tools using this interface: LDTP, dogtail, Accerciser.

Test scripts for LDTP, dogtail or Accerciser can be written by hand, possibly with assistance of recording tools to speed up the process. These scripts will be run individually or chained together from an automated procedure that can start with initiating a fresh test environment (such as copying and booting a VM instance).

Using an emulator designed to tunnel input to the running instance (such as Faumachine - http://www.faumachine.org) would be possible, but means that it will not be practical to run the same tests on real hardware separately.

The test cases should concentrate on integration and functional aspects. Detailed application specific tests should come from upstream developers, as part of the code. Running them, however, should be possible.

Implementation

  • Experiment with the various ATK-based tools, Faumachine, and other approaches, to find out which one would be the best one.
  • Pick one approach and implement the desktop tests in ["Testing/Cases"] with it. Skip the tests that require network access. Implement a "test runner" which can execute each test script and verify that the result is acceptable. For the ATK-based tools, the scripts would be in Python, and success would be measured by a) the script finishing without errors and b) optionally running a verification function (to check that a deleted file ends up in .Trash, for example).
  • Select tests from [:Testing/FixesToVerify:FixesToVerify] will be written during hardy in cases where they are fairly simple to automate.

Test/Demo Plan

Outstanding Issues

  • Programs are not usually designed to be tested this way. That results in recording tools working badly. Many widgets do not have names, for example. If this becomes an issue, bugs will be filed with upstream, possibly with patches.
  • Sharing tests with upstream developers and other distributions would be a good thing, both ways. Novell, for example, has test cases for Evolution and perhaps other programs as well. Novell uses LDTP. It may be necessary to work together with others to develop a culture for encouraging tests to be written, and for applications to be designed to be tested.

Notes

  • Using ATK currently means that only the GTK based Ubuntu flavors can be tested. KDE4 is said to get an accessibility layer as well, but it is not yet known to us whether that will be compatible with ATK or not.
  • Recording tools used for this project could also be used by users to record scripts to show bugs.

QATeam/Specs/DesktopAutomatedTests (last edited 2008-08-06 16:40:41 by localhost)