Testability

Testability is a UX testing framework, whose core feature for us is that it allows inspection of the tree of QObjects in a Qt application to read (write?) properties and call signals, as well as faking mouse/keyboard/gesture inputs, grabbing visual outputs, and measuring performance.

The Testability suite is comprised of the following packages:

* testability-qttas

• Contains a plugin that the Qt-based software under test (SUT) loads when you supply the “-testability” switch to allow deep introspection and interaction of the application, and a server binary (qttasserver) that exports this data to a client. Also sends fake events to the SUT as directed by the client.

* ruby-testability-driver

• Ruby library offering a basic client interface to qttasserver, enabling the SUT actions and inspection to be scripted

* ruby-testability-driver-qt-sut-plugin [Depends on ruby-testability-driver]

• Extension of the above Ruby library to add QObject inspection support to the client -- Qt-based SUT only.

* testability-visualizer

• GUI to allow easy inspection of SUT and writing tests [tdriver_visualizer]. This is not needed for running tests.

Installation Instructions

Testability is designed as a client/server system, so that the System under Test has as little changes as possible. However you can have both client and server on one machine.

On the System Under Test

Testability is not yet packaged in the official Ubuntu repositories, but is available in a PPA. Install it with the following steps:

sudo add-apt-repository ppa:gerboland/testability
sudo apt-get update

Install required packages with the command:

sudo apt-get install ruby-testability-driver testability-qttas xdotool xsel x11-utils wmctrl imagemagick

The additional utilities are required for generating application-external X events (mouse & keyboard) and performing window management.

To get raw log output from qttasserver, you should create the directory /logs/testability [FIXME!!]

On the Host

This machine runs the test suite, sending commands to qttasserver on the System Under Test and receiving the application state and other data.

If the host and target are different machines, you must add the PPA to it:

sudo add-apt-repo ppa:gerboland/testability
sudo apt-get update

Then install required packages for the host:

sudo apt-get install librmagick-ruby1.8 rubygems ruby-testability-driver \
ruby-testability-driver-qt-sut-plugin testability-visualizer 

The host needs to be configured with the IP address of the target. This involves editing /etc/tdriver/tdriver_parameters.xml to contain:

<parameters>
   <sut id="sut_qt" template="qt">
       <!-- use default values -->
       <parameter name="qttas_server_ip" value="127.0.0.1" />
   </sut>

   <!-- overload default behaviours parameter (see generic.xml in defaults folder) -->
   <parameter name="behaviours" value="behaviours.xml"/>
</parameters>

Note the line:

<parameter name="qttas_server_ip" value="127.0.0.1" />

The IP address supplied needs to match that of the System Under Test (aka target). If the target and host are on the same machine, use 127.0.0.1 as is done here.

Running the Tests on installed Unity-2d **PRECISE ONLY**

Here we describe obtaining test scripts from the unity-2d repository, and using them to test the installed version of Unity 2D.

Note that some of these tests may fail, as the tests in the repository are updated to reflect code changes.

Commands to run are

qttasserver &>/dev/null &
bzr branch lp:unity-2d
cd unity-2d/tests
ruby run-tests.rb -report

Then wait as the tests execute on your installed Unity-2d.

This will generate a reports directory in tests/tdriver_reports with a HTML and JUnit XML report detailing the results of the tests.

Testability Driver Guides

Quick guide to the Visualizer: https://projects.forum.nokia.com/Testabilitydriver/wiki/FeatureVisualizer

Guide to writing tests inside Visualizer: https://projects.forum.nokia.com/Testabilitydriver/wiki/FeatureVisualizerCodeEditor

Additional guides for more advanced abilities https://projects.developer.nokia.com/Testabilitydriver/wiki/TDriverHowTos

Reporting Bugs in the Test Suite

Explain briefly how to report bugs for your STP:

• Which project do you want testers to file the bugs agains? Ubuntu? Some other project?

• Link to How To Report Bugs in Ubuntu (https://help.ubuntu.com/community/ReportingBugs)

• Link to any specific debug information for your domain (https://wiki.ubuntu.com/DebuggingProcedures)