Ubuntu Wiki

Dev Week -- Automated Testing for the Desktop - Ara Pulido -- Thu, Sep 4

Ok, it is 4pm UTC, I think that we can start with the session
(12:01:19 PM) ara: Hello and welcome to the Automated Desktop Testing session, part of the Ubuntu Developer Week. Thanks for coming
(12:01:38 PM) ara: who is here for the automated test session?
(12:02:49 PM) ara: mmm, not many hands up...
(12:03:35 PM) ara: ok, anyway, we should start
(12:04:13 PM) ara: My name is Ara and I have started the Ubuntu Desktop Testing project
(12:04:28 PM) ara: (http://launchpad.net/ubuntu-desktop-testing)
(12:04:56 PM) ara: the project aims to create a nice framework to run & write desktop tests for Ubuntu
(12:05:31 PM) ara: If you have any questions during the session, please ask them in #ubuntu-classroom-chat, prefixed with "QUESTION: ..."
(12:05:38 PM) ara: so I can spot them quickly
(12:06:08 PM) ara: Also, if you don't understand something or you think I am going too fast, please, stop me at anytime
(12:06:50 PM) ara: let's start with a brief introduction to desktop automated testing, just in case you don't know what this session is about
(12:07:11 PM) ara: With automated desktop testing we name all the tests that runs directly against the user interface (UI), just like a normal user would do
(12:07:44 PM) ara: In GNOME this can be done by accessing the accessibility layer (AT-SPI)
(12:08:14 PM) ara: this layer was originally written for assistive technogies, like screen readers and such
(12:08:35 PM) ara: but it works pretty well for desktop automated testing
(12:09:02 PM) ara: that is why some frameworks use the AT-SPI layer to get access to the UI objects, get some information from them, and get them to do things (push buttons, write text, etc.).
(12:09:54 PM) ara: if you want to be able to run the examples during the session you would need to enable the assistive technologies
(12:10:11 PM) ara: and you must use GNOME, as the layer does not work for KDE
(12:10:23 PM) ara: Here they are some instructions on how to do it: https://wiki.ubuntu.com/Testing/Automation/LDTP#How%20to%20enable%20GNOME%20Assitive%20Technologies
(12:11:01 PM) ara: the bad news is that you would need to restart your gnome session if you want the changes to be applied
(12:11:29 PM) acemo: ara: will this session be learnfull even while am on a mac and don't have a linux pc near me atm?
(12:12:33 PM) ara: acemo: yes, you can stay anyway :)
(12:12:39 PM) dholbach: acemo: do you think you can ask your next question in #ubuntu-classroom-chat please and prefix it with QUESTION:? :)
(12:12:40 PM) ara: acemo: please, post questions at -chat
(12:12:49 PM) ara: <dholbach> QUESTION: How about XFCE, which parts of GNOME are required? :)
(12:13:46 PM) ara: dholbach: I think assistive technologies are present in xfce, but i haven't tested the ubuntu desktop testing with it yet
(12:14:23 PM) dholbach: OK
(12:14:31 PM) ara: For the Ubuntu Desktop Testing layer we are using LDTP (http://ldtp.freedesktop.org/), that has a python library for writing tests. This is one of those automated desktop testing frameworks that use the at-spi layer
(12:14:59 PM) ara: When using this library you have to use some specific information from the UI in order to recognize the objects (window titles, object hierarchy, etc)
(12:16:13 PM) ara: I.e. if you want to click a button in the Gedit window, first you will need to recognize the window, then obtain its children, and finally click the selected button.
(12:16:38 PM) ara: If we add all that information to the script and then the UI changes, we would need to change all the scripts to match the new UI changes.
(12:18:24 PM) ara: <pwnguin> QUESTION: so this all won't be very useful for testing mouse input tools like cellwriter
(12:19:13 PM) ara: pwnguin: I haven't tried cellwriter, but it should be ok. Mouse inputs can be mimic through the at-spi layer
(12:19:44 PM) ara: pwnguin: you won't be testing the mouse itself, obviously, you will be testing the tool
(12:20:24 PM) ara: One of the main objectives that we are persuading when creating a testing framework for Ubuntu desktop is to avoid scripts to know anything about the objects behind them.
(12:20:52 PM) ara: Definitively, these objects will still require to be maintained, but the logic of the scripts will remain the same.
(12:22:02 PM) ara: One example. Let’s imagine that we had a regression test suite for Gedit that will edit, modify, open and save several files.
(12:22:17 PM) ara: Many
(12:22:27 PM) ara: About a hundred
(12:22:39 PM) ara: If any of the Gedit features changes its UI, only the Gedit class will be modified. All the scripts will still be valid.
(12:24:17 PM) ara: The other good thing about it is that people willing to add new test cases to ubuntu, can do it easily
(12:24:50 PM) ara: which version of Ubuntu are you running at the moment?
(12:25:32 PM) ara: ok
(12:25:50 PM) ara: If you are using Intrepid you can install the desktop-testing-library easily through PPAs: https://wiki.ubuntu.com/Testing/Automation/LDTP/HowToUseTestingLibrary#Installation
(12:26:15 PM) ara: Hardy PPAs are also available, but they are not well maintained, therefore some things might be broken: https://wiki.ubuntu.com/Testing/Automation/LDTP/HowToUseTestingLibrary#Notes%20on%20Hardy%20Heron%20(Ubuntu%208.04)
(12:27:02 PM) ara: Don't worry, we won't perform any potentially harmful tests for this session >:-)
(12:28:12 PM) ara: <pwnguin> QUESTION: where should bugs against a PPA be reported?
(12:28:27 PM) ara: pwnguin: please, don't file them in Ubuntu project :-)
(12:28:43 PM) ara: pwnguin: ping me in the irc or use my email address :-)
(12:29:24 PM) ara: pwnguin: but you can use the LP project
(12:29:31 PM) ara: pwnguin: and file them in bugs
(12:29:42 PM) ara: https://launchpad.net/ubuntu-desktop-testing
(12:30:50 PM) ara: The Library API is up-to-date and it is available at: http://people.ubuntu.com/~ara/ldtp/doc/testing_module_doc/
(12:31:17 PM) ara: Right now we have classes for Gedit, Update Manager and GkSu. We also have a generic Application class to gather common behaviour from GNOME applications.
(12:31:38 PM) ara: <thekorn> Question: GUI testing is kind of new to me, but I have read about dogtail, how does dogtail fit into this concept, or is it totally different?
(12:31:46 PM) ara: thekorn: good question
(12:32:22 PM) ara: dogtail is another of those desktop testing frameworks that use at-spi layer to access the gnome objects
(12:32:48 PM) ara: dogtail is completely written in python, while LDTP is C+python
(12:33:13 PM) ara: python only would be easier to maintained, but the truth is that the LDTP upstream project is much more active, that the dogtail one
(12:33:37 PM) ara: that's the main reason we decided to go for ldtp
(12:34:08 PM) ara: <tacone> what's wrong with at-spi ? why not just using that ?
(12:35:16 PM) ara: tacone: simplicity. ldtp (or dogtail) makes it easier to write scripts
(12:35:46 PM) ara: hidding some low level assistive techology programming stuff
(12:36:37 PM) ara: <mnemo> QUESTION: If I write tests using "ubuntu desktop testing" can I still run those tests on Fedora/Debian??
(12:37:18 PM) ara: mnemo: you can, many of them will fail, though :) And you will need to install the testing library manually
(12:38:12 PM) ara: mnemo: but common stuff, like applications that run in GNOME and don't have many ubuntu tweaks will work
(12:39:25 PM) ara: let's see an example on the difference on writing tests for ubuntu using the testing library and using only LDTP
(12:40:06 PM) ara: This is the link to the code using the testing library: https://wiki.ubuntu.com/Testing/Automation/LDTP/HowToUseTestingLibrary/Comparison/UsingDesktopTestingLibrary
(12:41:19 PM) ara: As you can see the code is clean and almost self-explaining
(12:42:00 PM) ara: Now the code using pure LDTP code: https://wiki.ubuntu.com/Testing/Automation/LDTP/HowToUseTestingLibrary/Comparison/PureLDTPCode
(12:43:43 PM) ara: Now the code becomes less clear, with LDTP specific code and application constants dependent. Also the desktop testing library include error checking code that I have removed from this example to make it clearer
(12:44:45 PM) ara: QUESTION: how do you judge a test's pass or failure?
(12:45:42 PM) ara: pwnguin: that is something ldtp can hide. If something breaks in the application, an ldtpexcetpio is raised, which can be used for logging failures
(12:46:22 PM) ara: pwnguin: also, in the testing library, i am writing "check" class (part of the check module, see the api documentation) to check things a little bit more complicated
(12:46:48 PM) ara: pwnguin: i.e. comparing a gedit saved file against one that we know it looks like it should
(12:47:43 PM) ara: <tacone> see line 141 for the save function http://paste.pocoo.org/show/84369/
(12:47:51 PM) ara: <tacone> Question: do I have to write similar code for every new application I test, right ?
(12:48:32 PM) ara: tacone: yes, and no. if the application is simple, and saving only saves with the common dialogs, you can use the Application class instead, that also has a save method
(12:48:58 PM) ara: You can download this example and try it on your machine. The script is available at http://people.ubuntu.com/~ara/udw/gedit/intrepid.py
(12:49:06 PM) ara: I have added also a working script for hardy, just in case you want to try that on hardy http://people.ubuntu.com/~ara/udw/gedit/hardy.py
(12:49:22 PM) ara: Download the file and run
(12:49:26 PM) ara: python intrepid.py
(12:50:03 PM) ara: That should make the magic start (if you have enabled the assistive techonogies, and have the desktop testing library installed...)
(12:51:27 PM) ara: well, we are running out of time, let's wrap up
(12:52:09 PM) ara: You can contribute easily, with very little programming knowledge, to the automated testing efforts by writing new test scripts using the testing library. A How-To guide is available at https://wiki.ubuntu.com/Testing/Automation/LDTP/HowToUseTestingLibrary
(12:52:57 PM) ara: If you have any questions you can ping me in #ubuntu-testing channel or at my email address
(12:53:10 PM) ara: Also, if you have more advanced python knowledge and would like to give a try on extending the desktop library that would also be great
(12:53:53 PM) ara: please, bear in mind that we are focusing on intrepid now, so fixing bugs for hardy is not a priority :)
(12:54:30 PM) ara: <tacone> Question2: for what applications will you develop test cases?
(12:55:22 PM) ara: tacone: we would like first to add a lot of coverage to one or two main ubuntu applicaitons, like the update manager. not only the test cases, but mainly the library. so adding new test cases should be easy
(12:56:24 PM) ara: <pwnguin> QUESTION: is this project something you expect wider upstream participation in the distant future for?
(12:57:19 PM) ara: pwnguin: sure! We would like to make the gnome classes as much as ubuntu independent as possible, so they can be use in other distributions and/or upstream teams. but as you said, distant future ;-)
(12:57:30 PM) ara: Ok, no time for anything else
(12:57:44 PM) ara: Thanks everybody for joining in