Dev Week -- Automated Desktop Testing -- ara -- Thu, Jan 22
UTC -4 (EST)
(11:01:09 AM) ara: OK, it is 4pm UTC, I think that we can start with the session (11:01:27 AM) ara: thanks dholbach and everybody else for the warm welcome (11:02:05 AM) ara: as dholbach just said, I am going to talk about Automated Desktop Testing, but first let me tell you a secret (11:02:21 AM) ara: a secret very few developers know (11:02:28 AM) ara: it will change your life as you know it (11:02:35 AM) ara: ok, here it is: (11:02:51 AM) ara: testing software is *FUN* (11:03:18 AM) ara: not boring, not tedious, but FUN (11:03:53 AM) ara: my name is Ara and I am part of the Ubuntu QA Team (11:04:10 AM) ara: (https://wiki.ubuntu.com/QATeam/) (11:04:38 AM) ara: as part of my duties in the team I have started the Ubuntu Desktop Testing project (11:04:38 AM) ara: (http://launchpad.net/ubuntu-desktop-testing) (11:05:10 AM) ara: The project aims to create a framework to run & write automated desktop tests for Ubuntu (11:05:49 AM) ara: To be able to try some of the tools that I am going to present, you will need to have installed 'bzr' in your computers. If you don't have it, you can install it easly: (11:05:49 AM) ara: $ sudo apt-get install bzr (11:06:20 AM) ara: Also, if you don't understand something or you think I am going too fast, please, please, please, stop me at anytime (asking in the -chat room) (11:06:52 AM) ara: Ape3000: QUESTION: Should I use my Intrepid desktop or Jaunty virtualbox machine? (11:07:04 AM) ara: Ape3000: Both of them should work (11:07:31 AM) ara: Let's start with a brief introduction to desktop automated testing, just in case you don't know what this session is about (11:07:56 AM) ara: With automated desktop testing we name all the tests that runs directly against the user interface (UI), just like a normal user would do (11:08:58 AM) ara: a script will run and you will start to see buttons clicking, menus poping up and down and things happening, automagically (11:09:15 AM) ara: Ireyon: does that answer your question? (11:09:29 AM) Ireyon: yes it does, thanks (11:09:41 AM) ara: In Ubuntu we do this by accessing the GNOME accessibility layer (AT-SPI) (11:10:03 AM) ara: This layer was originally written for assistive technologies, like screen readers and such (11:10:30 AM) ara: technologies to make computers accessible to people with disabilities (11:10:51 AM) ara: but it turned out that it works pretty well for desktop automated testing (11:11:22 AM) ara: that is why some testing frameworks use the AT-SPI layer to get access to the UI objects, get some information from them, and get them to do things (push buttons, write text, etc.). (11:12:05 AM) ara: if you want to be able to run the examples during the session you would need to enable the assistive technologies (11:12:05 AM) ara: and you must use GNOME, as the layer does not work for KDE (11:12:05 AM) ara: Here they are some instructions on how to do it: https://wiki.ubuntu.com/Testing/Automation/Desktop/#How%20to%20run%20the%20tests (11:12:49 AM) ara: basically is checking the box in the dialog in System->Preferences->Assistive Technologies (11:13:41 AM) ara: the bad bad bad news is that you would need to restart your gnome session if you want the changes to be applied (if you are using a virtual machine for the examples this is not as bad news) (11:14:12 AM) ara: weboide: QUESTION: Can those tests make the machine become unstable/frozen/broken? (11:15:10 AM) ara: weboide: depending on what you do with your tests :-) normally they do no harm, but the at-spi layer still have some bugs, so you may spot some of thems (11:15:36 AM) ara: For the Ubuntu Desktop Testing project we are using LDTP (http://ldtp.freedesktop.org/), that has a python library for writing tests. This is one of those automated desktop testing frameworks that use the at-spi layer (11:16:01 AM) ara: If you want to run the examples in this session, please, install LDTP: (11:16:09 AM) ara: $ sudo apt-get install python-ldtp ldtp (11:16:43 AM) ara: ia: QUESTION: does exist some log system in this tests, so user/developer after testing could see, what goes well and what goes wrong? (11:17:21 AM) ara: ia: sure! you don't have to watch your tests as they are running. You can go and do your things and come back to check the results. We will be seeing that later on (11:18:06 AM) ara: When using this library you have to use some specific information from the UI in order to recognize the objects (window titles, object hierarchy, etc) (11:18:31 AM) ara: I.e. if you want to click a button in the Gedit window, first you will need to recognize the window, then obtain its children, and finally click the selected button. (11:19:06 AM) ara: If we add all that information to the script and then the UI changes, we would need to change all the scripts to match the new UI changes. (11:19:49 AM) ara: One of the main objectives that we are trying when we create a testing framework for Ubuntu desktop is to avoid scripts to know anything about the objects behind them. (11:20:25 AM) ara: Definitively, these objects will still require to be maintained, but the logic of the scripts will remain the same. (11:20:25 AM) ara: One example. Let’s imagine that we had a regression test suite for Gedit that will edit, modify, open and save several files. (11:21:00 AM) ara: If any of the Gedit features changes its UI, only the Gedit class will be modified. All the scripts will still be valid. (11:21:01 AM) ara: The other good thing about it is that people willing to add new test cases to ubuntu, can do it easily (11:21:25 AM) ara: they don't need to know much about ap-spi or LDTP, just some basic python scripting will do (11:21:44 AM) ara: If you're running Intrepid or Hardy you can get the Ubuntu Desktop Testing library as: (11:21:54 AM) ara: $ bzr branch lp:ubuntu-desktop-testing/intrepid ubuntu-desktop-testing (11:22:06 AM) ara: If, by any chance, you're running Jaunty, the trunk fixes some differences in this release: (11:22:06 AM) ara: $ bzr branch lp:ubuntu-desktop-testing (11:22:44 AM) ara: If you find any bugs in the UDT library itself, please, report them in the Launchpad project: (11:22:44 AM) ara: https://launchpad.net/ubuntu-desktop-testing (11:22:55 AM) ara: don't report them to the Ubuntu project :-) (11:23:18 AM) gman_ is now known as gman16k (11:23:23 AM) ara: The Library API is up-to-date and it is available at: http://people.ubuntu.com/~ara/ubuntu-desktop-testing/doc/ (11:23:23 AM) ara: Right now we have classes for Gedit, Update Manager and PolicyKit and Seahorse. We also have a generic Application class to gather common behaviour from GNOME applications. (11:24:11 AM) ara: Let's see an example on the difference on writing tests for ubuntu using the testing library and using only LDTP (11:24:40 AM) ara: so you can understand better what I mean with "basic python scripting" (11:25:07 AM) ara: This is the link to the code using the testing library: https://wiki.ubuntu.com/Testing/Automation/Desktop/HowToUseTestingLibrary/Comparison/UsingDesktopTestingLibrary (11:25:27 AM) ara: what would you say this code does? (11:26:35 AM) ara: weboide, Ireyon: yes. that's it (11:26:49 AM) ara: the code is more or less self explanatory (11:26:55 AM) ara: (and commented, just in case...) (11:27:34 AM) ara: Now the code using pure LDTP code, with out the Ubuntu Desktop testing library: (11:27:35 AM) ara: https://wiki.ubuntu.com/Testing/Automation/Desktop/HowToUseTestingLibrary/Comparison/PureLDTPCode (11:29:21 AM) ara: It is not very very complicated, but the code is becoming less clear, and more difficult to understand for new people (11:29:39 AM) ara: Also the desktop testing library include error checking code that I have removed from this example to make it clearer (11:29:56 AM) ara: not to say that using at-spi code directly would make things even harder... (11:30:21 AM) ara: Ok, let's try a small example. Remember that you will need to have the Assistive Technologies enabled to try this at home :-) (11:31:10 AM) ara: if you don't have them enabled, you can log out and come back (I will answer questions in the next 5min if people decide this) or wait and try it when the session finishes (11:31:20 AM) ara: what do you guys prefer? (11:32:55 AM) ara: Ireyon: I don't get the libs installed (11:32:59 AM) ara: Ireyon: what libs? (11:33:10 AM) Ireyon: the python libs (11:33:27 AM) ara: Ireyon: what error do you get? (11:33:58 AM) ara: ok, we will go on (11:34:04 AM) ara: The example that we are going to run will open gedit, will write some characters in there, will save the file, and will compare it to another one to check for pass/fail. Be sure to close all your gedit sessions before running this test, to avoid messing up your documents. (11:34:44 AM) ara: Let the magic start: (11:34:56 AM) ara: $ cd ubuntu-desktop-testing (11:34:56 AM) ara: $ ../bin/ubuntu-desktop-test -a gedit (11:35:05 AM) ara: change that last one by (11:35:14 AM) ara: $ ./bin/ubuntu-desktop-test -a gedit (11:35:15 AM) ara: sorry (11:36:53 AM) ara: OK, once that your test has finished, you can grab the logs (11:36:57 AM) ara: ia: ^ (11:37:14 AM) ara: the logs can be found at ~/.ubuntu-desktop-tests/gedit (11:37:41 AM) ara: you will find and XML log (in case you want to transform it to something else) (11:37:53 AM) ara: and a nice HTML report, in case you want to publish that somewhere (11:38:11 AM) sdx24 is now known as sdx23 (11:39:16 AM) ara: Ireyon: LdtpExecutionError: "Mmm, something went wrong when saving the current document:'The mnuSave menu was not found.'" (11:39:36 AM) ara: Ireyon: did you close the window? before the test finished? (11:40:20 AM) ara: Ape3000: QUESTION: Would it be possible to create desktop macros for general use with this? (11:40:48 AM) ara: Ape3000: yes, you could. I don't think that it is the best library for macros, though, but yes, you could use it for that (11:42:07 AM) ara: one of the classic questions that people ask is, can I use this to test Fedora/OpenSuSE/Debian/OtherDistroOfMyChoice... (11:43:12 AM) ara: the answer is yes (11:43:34 AM) ara: and the good thing is that it is the perfect timing if you want to join a brand new team (11:43:54 AM) ara: The GNOME people started to be really interested in the project, so we decided to create a GNOME desktop testing project (11:44:04 AM) ara: (https://launchpad.net/gnome-desktop-testing) (11:44:18 AM) ara: The code is now in LP, but it will move eventually to GNOME SVN servers. (11:44:36 AM) ara: It includes the things in ubuntu-desktop-testing that can be apply to all the GNOME environments (11:45:00 AM) ara: i.e. it does not include UpdateManager class, because that it is something Ubuntu exclusive (11:45:32 AM) ara: If you'd like to contribute and be part of this motivating new project, go ahead and show your love at http://live.gnome.org/DesktopTesting (11:46:16 AM) ara: Ape3000: QUESTION: So what are the best things with using this kind of automation? (11:46:49 AM) ara: Ape3000: regression testing, mainly. If something breaks between releases, those tests can be useful to catch this kind of regressions (11:48:12 AM) ara: This technology can be used also to test Xubuntu (11:48:30 AM) ara: so, if you are using Xubuntu and want to contribute, that would be great (11:48:51 AM) ara: right now we are only giving coverage to Ubuntu (GNOME desktop) (11:49:31 AM) ara: Ok, only ten minutes left, let's wrap up (11:49:50 AM) ara: Ape3000: QUESTION: Do you have any example bugs / cases where automated desktop testing was helping greatly? (11:50:30 AM) ara: Ape3000: The project is still very young, but we helped solving some accessibility bugs, while creating our tests (11:50:38 AM) ara: Ape3000: a list https://wiki.ubuntu.com/Testing/Automation/AtspiBlockers (11:51:25 AM) ara: You can contribute easily, with very little programming knowledge, to the automated testing efforts by writing new test scripts using the testing library. A How-To guide is available at https://wiki.ubuntu.com/Testing/Automation/Desktop/HowToUseTestingLibrary (11:51:52 AM) ara: Also, if you have more advanced python knowledge and would like to give a try on extending the desktop library that would also be great (11:52:12 AM) ara: Please, apply your changes in your branch and use "Merge proposal" feature in Launchpad! (11:52:42 AM) ara: Any other questions? (11:53:10 AM) ara: counting down... (11:53:13 AM) ara: 5... (11:53:17 AM) ara: 4... (11:53:20 AM) ara: 3... (11:53:23 AM) ara: 2... (11:53:28 AM) ara: 1... (11:53:33 AM) ara: ... (11:53:49 AM) ara: loic-m: QUESTION: Is it possible to use this for benchmarking desktop tasks (i.e. à la Phoronix)? (11:54:00 AM) ara: (with the bell) (11:54:17 AM) ara: loic-m: Yes. It is not specifically a benchmarking framework (11:54:36 AM) ara: loic-m: but times are always taken between tests (11:55:02 AM) ara: loic-m: and it is pure python code, so you can use any python library of your choice (11:55:46 AM) ara: no more questions? (11:57:04 AM) ara: [srx]: QUESTION: can we print like this? (11:57:13 AM) ara: [srx]: I don't understand the questions, sorry (11:57:42 AM) ara: One more thing. These tests work only (for the moment) if you have your desktop in English :-) (11:57:44 AM) ara: Ireyon: ^ (11:58:05 AM) ara: QUESTION: is this only for Gnome applications, or will it work for qt/tcl-tk/etc? (11:58:33 AM) ara: loic-m, StyXman: KDE is working on having a better accessibility layer. but for the moment these tests only work for GNOME (11:59:01 AM) iamarockstar: i guys (11:59:16 AM) ara: Ireyon: QUESTION: Is there any chance to get it working with any language? (11:59:24 AM) iamarockstar: what is this room for i am new here (11:59:55 AM) ara: Ireyon: right now we are starting giving coverage to English desktop. Maybe in the future... (12:00:10 PM) ara: OK. No time for more. If you have any questions you can ping me in #ubuntu-testing channel or at my email address <ara AT ubuntu.com> (12:00:27 PM) ara: Thanks all for coming! I hope you all enjoyed the session!