MagoDesktopTests
Dev Week -- Let Mago do your Desktop testing for you -- ara -- Thu Sep 3rd, 2009
Hi! and welcome everybody!
My name is Ara and I am part of the Ubuntu QA Team
I am a software tester and I love testing. I always try to convince devs about testing being something fun :-). As part of my duties in the QA team I have started the Mago project but, what's Mago?
I like to call Mago a desktop testing "initiative", rather than a framework. In fact, it is heavily based on Linux Desktop Testing Project (LDTP), a desktop testing framework, written in C.
With automated desktop testing we name all the tests that runs directly against the user interface (UI), just like a normal user would do: a script will run and you will start to see buttons clicking, menus poping up and down and things happening, automagically. Mago tries to add consistency to the way we write, run and report results of this kind of scripts.
The aim of this session is to present you this "initiative" and the way we do things in Mago. As stated at https://wiki.ubuntu.com/UbuntuDeveloperWeek/Sessions, this session required (at least it was very very recommended) to follow http://mago.ubuntu.com/Documentation/GettingStarted beforehand. Who did follow it? Please, answer in the -chat channel or who didn't? 
Okeeey, don't worry, you can still attend and follow the session if you haven't done your homework. If you haven't followed the getting started guide, please, do the following:
$ sudo apt-get install python-ldtp bzr $ bzr branch lp:mago
That will install the LDTP packages and the BZR package to be able to get the mago source code
If you don't understand something or you think I am going too fast, please, please, please, stop me at anytime (asking in the -chat room). I will try to follow the -chat channel and answer your questions as we go by. So, let's dive in.
With Mago, one of the things that we are building is a library with "testable" applications. If the application you want to test is already in the mago library, writing tests for it is easier. We will start from there, and if we have time then we can start on how to add new applications to the library.
First, some testing terminology: A test suite is a collection of test cases; with a test case being an scenario you want to test in your application.
We also need to be able to determine whether a test is successful or not in order to report pass or fail. The "knowledge" we have that let us do that is an "oracle"
<frandieguez> QUESTION: Mago is like Rspec or TestUnit ara?
No, those are unit test frameworks. Mago is for testing the UI directly You don't need to have or to know the code of the application you're about to test
In Mago a test suite consists of 2 files: a PYTHON file and a XML file. The .py file contains the code of the test. The things you want to do with the application. The .xml file is the description and arguments of the test suite. This file makes the .py file reusable in different test cases. Let's see how. In your created mago folder (branching the code), open the gedit folder. We keep test suites ordered by application, to allow running test suites for only one of the applications.
Under the gedit folder you have gedit_chains.xml and a gedit_chains.py. They both have the same name, but this is not necessary. Open the Python file with your preferred text editor.
<efm> does it matter if the app is gnome or kde or something else?
Right now, desktop testing is based in accessibility information, and kde is very poor. Right now you need to be running Gnome in order to test the application. That's going to change in a future, when at-spi (the communication layer for accessibility) gets ported to DBUS.
So, going back to the python file... The python file is only a simple python class inheriting from GeditTestSuite, which also inherits from TestSuite. All Mago test suites are classes that inherit, directly or inderectly from TestSuite. The main part:
if __name__ == "__main__":
gedit_chains_test = GEditChain()
gedit_chains_test.run()is not necessary, because Mago will run the tests for you, but you can add it to your code for testing purposes.
<arky> QUESTION: Can mago be used to discover a11y problems in some apps, like missing description or invalid relationships <arky> well, it is the other way round, if an application has poor a11y information, it is going to be difficult to test (or impossible)
But there are better ways to test if an application has good a11y information, like using Accerciser.
So, we have a class, GEditChain, which contains a method. A test suite can contain as many methods as wanted. The only test method, "testChain", opens the application, writes on the first tab the string passed as the "chain" argument; saves it and compares the saved file with an oracle file. (again oracle, in testing, means the "right" thing a test has to do. Something we know before hand that it is right, so we can check if our test is correct), and then closes the application.
As you can see in the code, there is not such a thing as "open" or "close" the application. This is done by the test suite for you. Mago leverage those sort of things, so you can concentrate on your test case. We will get back to that afterwards.
Now open the XML file with your preferred text editor. As you can see, it is a simple XML file. The root node, called "suite" allows setting a name for your suite. In this case, "gedit chains". The first child node, class, determines the python class of that test suite. In our case the class is the one we saw before.
After the class node, we have a node call description. This is a text description of the suite and it will be included in the reports for your convenience. If you want your reports to be self explanatory, you have to include a nice description here 
After that, there are as many "case" nodes as test cases included in the test suite. In our case we have two cases: "Unicode Tests" and "ASCII Tests". This is one of the advantages of separate the description and data, from the actual script. We can easily reuse the method for several test cases.
Each case has a "method", testChain in the example, a description, which will also be included in the report, and a set of arguments. These arguments need to match the arguments in the method.
So, let's try to run these tests using mago. If you are running any gedit session, please, close it if you don't want to lose your work
Go back to your mago folder and run:
$ PYTHONPATH=. ./bin/mago -f gedit_chains
That will run all the test cases in a test suite file name called gedit_chains. Once finished, you can check the test logs under ~/.mago/gedit.
Under that folder, Mago have created two log files: the .log file is an XML, in case you want to parse for something else. The .html is a nice HTML report, with screenshots if something went wrong.
OK, there are a couple of questions in the -chat channel about being a bit slow
Mago is based on LDTP, which uses c-spi, a slow, slow library. This is going to change, because LDTP2 is being finished now, based in pyspi, and much faster.
OK, let's continue by adding a test case for the same method. Open again the XML file and let's edit it. We will add it after the last one. Add the node at http://paste.ubuntu.com/264357/ Before, the </suite> one, of course.
You can see the objective of the test case: open the application, write a text "Happy Ubuntu Developer Week!", save it, and compare it to the oracle. We have to write the oracle file beforehand, so open a text editor, create a file with the text "Happy Ubuntu Developer Week!" (without no new lines) and save it as "udw.txt" in the gedit/data folder.
While you do this, I'll answer another question.
<tedg> QUESTION: Is it possible to run unattended on a head-less server? Like without X?
tedg, I am afraid you can't. A full GNOME session is needed tedg, not only X, but also a gnome session
Done? The udw.txt file, I mean. So lets's run again the test, always from the mago root folder:
$ PYTHONPATH=. ./bin/mago -f gedit_chains
This time, a new test case is also run for you, which will compare the new string to the newly created file.
- QUESTION: ara, how you can execute just one test case?
There is an option in mago to do that. run } to check it
OK, let's take it to the next level. Let's think that we want to do the opposite test case, opening a file, reading its contents and compare to the string we know it contains. Let's create, under the gedit folder, a new file gedit_open.py with the following code: http://paste.ubuntu.com/264358/
Also, let's create an XML file to run this (gedit_open.xml) http://paste.ubuntu.com/264362/
In this case, the oracle is the string that we know the file contains. Let's try to run this: PYTHONPATH=. ./bin/mago -f gedit_open
The application opened, mago gave an error, and exited the application. That's expected, though. The GEdit class does not contain a openfile method. We need to use LDTP functions to add new methods to the GEdit class. As we said at the beggining, one of the aims of Mago is reuse. Right now GEdit does not include a openfile method, but once added, anyone can benefit from this addtion and use the method easly in their test scripts.
The GEdit class is under the mago library, application, gnome.py Lets open it: $ <editor> mago/application/gnome.py Search for "class GEdit" and let's start editing.
Don't worry about LDTP syntax, that's another story. In this session we want to learn about the internals of Mago and how to contribute to it. LDTP has its own documentation and tutorials at http://ldtp.freedesktop.org/wiki/Docs Going back to Mago. Mago library contains a set of resuable methods for testing applications. We want to avoid having ldtp functions in our scripts, and leave that in the library. If anything changes in the application, or we decide to change the framework, the scripts will remain the same.
Let's add these two methods to the library: In the GEdit class, add the two methods at http://paste.ubuntu.com/264361/ We are adding a method to open a file in Gedit, and another to get the contents of the main buffer. All strings, as per Mago coding guidelines, should be set as constants of the class. Check the rest of the methods for an example. For the sake of simplicity of this tutorial, we have kept those as strings in the code.
So you can start thinking about how LDTP recognizes the objects in an application.
OK, let's save the file and let's try to run it one more time now: PYTHONPATH=. ./bin/mago -f gedit_open How did it work this time?
I am afraid we won't have time to cover other topics, like adding a new application to the mago library, but before we finish the session I would like to talk you briefly about how the magic of opening the applications and closing them works. As a told you, the GEdit test suite that we created, inherits from GEditTestSuite, which inherits itself from SingleApplicationTestSuite. Let's see what a TestSuite class and subclasses need to implement: $ <editor> mago/test_suite/main.py Every TestSuite class and subclasses need to reimplement, if needed, the setup, the teardown and the cleanup methods. The setup method is run before running any of the test cases, the clean up after every test case, and teardown, after the whole suite is run.
Let's take a look to the GEditTestSuite class: $ <editor> mago/test_suite/gnome.py What we do on the setup is opening the application. That's obvious. We close the application on the teardown method.
The most complicated one in this case, is the cleanup method, run between test cases. In this one we close the current gedit tab, ignore a "Save file" dialog if it appears, and create a new document; leaving gedit again, clean and ready for the next test case. If you get errors about the setup, cleanup or teardown methods, it is here where you have to start debugging.
I am running out of time to try to help solving the errors you got, you can catch me anytime at ara AT ubuntu DOT com We can finish here leaving you with some documentation in case you want to go deeper: http://mago.ubuntu.com It has all the information you need: mailing list, IRC channel, API doc, etc.
I really recommend joining the mailing list: you can add there your errors when writing test cases, and the community is always happy to help Thanks all for attending and happy testing!!