Ubuntu Wiki

Dev Week -- Writing Scripts For Automated Desktop Testing -- Lars Wirzenius -- Thu, Feb 21

(12:01:55 PM) liw: I'm going to lead a session on writing scripts for automated desktop testing
(12:02:26 PM) liw: I would like to have a volunteer to relay questions from the -chat channel so that I don't have to do that at the same time as talking here -- anyone?
(12:02:56 PM) liw: but let's start
(12:03:17 PM) liw: the first part of this session is going to be about running existing scripts
(12:03:25 PM) Grazieno: liw: I can do it
(12:03:32 PM) liw: Grazieno, thank you!
(12:04:13 PM) liw: there is going to be a nice framework to run the tests automatically, but that isn't done yet, so we're going to concentrate on the scripts themselves, running them manually and writing them
(12:05:11 PM) liw: we're using the AT-SPI framework for the scripts; it was originally written for accessibility, but provides a way for external programs to poke GUI applications and get them to do things, and to examine some of their internal state
(12:05:48 PM) liw: AT-SPI only works in GNOME, not in KDE (and may work in Xfce, but I haven't investigated)
(12:06:19 PM) liw: to enable it, go to System -> Preferences -> Universal Access -> Assistive Technology, then tick the "Enable..." checkbox
(12:06:38 PM) liw: you may want to set up a dedicated account for this, though
(12:06:48 PM) liw: or run things in kvm, or virtualbox, or another emulator
(12:07:11 PM) liw: because there's a fair chance that the scripts don't work they way they're supposed to work, and they may then break things
(12:08:29 PM) liw: I use kvm myself, and I am making a Qemu image file (which kvm can use) that has everything already set up to run the tests, but that's still uploading (it's amazing how long it can take to upload big files), so I won't point you at the url yet
(12:09:08 PM) liw: kvm is basically qemu, but with hardware acceleration; you can run the following command to see if your CPU can run kvm: egrep '^flags.*(vmx|svm)' /proc/cpuinfo
(12:09:56 PM) liw: if that returns anything, and you want kvm, upgrade to hardy, install the kvm package, do "adduser $LOGIN kvm" (where $LOGIN is your username), and log out and log back in again
(12:10:35 PM) liw: oh, and I forgot: if you want to enable the assistive technologies stuff, you need to log out and back in again as well, it's not enough to just tick the checkbox (the AT-SPI services need to be started when you log in)
(12:11:05 PM) liw: anyone here already have an environment where they have assistive technology enabled?
(12:11:09 PM) Grazieno: liw: ok, I'll be back :)
(12:11:13 PM) Grazieno left the room ("Ex-Chat").
(12:11:30 PM) liw: if anyone else wants to enable it, now would be a good time, I can wait a few minutes
(12:12:21 PM) Grazieno [n=gate@200.128.80.254] entered the room.
(12:12:55 PM) Grazieno: liw: ready!
(12:13:47 PM) liw: https://wiki.ubuntu.com/Testing/Automation/DogtailTutorial has instructions for enabling AT and is generally useful for this automated testing stuff, please bookmark
(12:14:02 PM) liw: ok, next step is to install the python-dogtail package
(12:14:11 PM) liw: (please do so now  if you want to experiment today)
(12:14:31 PM) liw: also, we're going to be using a sample script for gThumb, so install that, too, if you don't have it already
(12:15:55 PM) liw: https://wiki.ubuntu.com/Testing/Automation/Coverage has some scripts that have been written and collected -- there's way too few of them, since I've been tinkering with the framework to run them instead of writing new scripts, but let's hope some people here will get excited and help with the script writing (which is the more interesting part of the job) :)
(12:16:35 PM) liw: eddyMul, just in case you missed things: https://wiki.ubuntu.com/Testing/Automation/DogtailTutorial is a useful link, and if you want to play with things tonight, install python-dogtail and gthumb packages
(12:17:12 PM) eddyMul: liw: thanks
(12:17:36 PM) liw: those wanting to try running a script, download the gthumb.tar.gz file from the Coverage page (url above) and unpack it
(12:18:03 PM) liw: then, from the command line, enter the directory with the unpacked files, and run this command: LC_ALL=C python gthumb-preferences.py
(12:18:12 PM) liw: magical things should now be happening
(12:18:24 PM) liw: is anyone seeing magic?
(12:18:47 PM) liw: or is anyone trying to see magic and failing?
(12:19:35 PM) Grazieno: liw: failing..
(12:19:40 PM) charliecb: hi
(12:19:46 PM) liw: Grazieno, what's the error?
(12:20:19 PM) Grazieno: liw: http://paste.ubuntu.com/4851/
(12:20:50 PM) liw: Grazieno, weird... are you running this in hardy?
(12:21:12 PM) Grazieno: liw: yes :)
(12:21:28 PM) Grazieno: liw: wrong?
(12:21:33 PM) liw: Grazieno, hardy is right
(12:22:09 PM) eddyMul: liw: gutsy is wrong?   :(
(12:22:25 PM) liw: eddyMul, I know hardy is right, but I don't know if gutsy will work or not
(12:22:38 PM) eddyMul: liw: failed for me: http://paste.ubuntu.com/4852/             :(
(12:23:27 PM) liw: ok, this is a bit disturbing, but I think I'll want to continue with the session -- I'd like to help you guys get this working, but we can perhaps do that later?
(12:23:31 PM) liw: is that ok?
(12:23:49 PM) Grazieno: liw: yes!
(12:24:28 PM) eddyMul: liw: yes. carry on.
(12:24:37 PM) liw: I should'v emade a movie about how the gthumb-preferences.py script runs and opens gthumb and toggles various preferences
(12:24:53 PM) liw: you'll have to just believe me that it does...
(12:25:26 PM) liw: the crucial bit here would have been to get it to work so that we can progress to writing new scripts
(12:25:39 PM) liw: but well, we'll fix that later
(12:25:49 PM) Grazieno: <jimcooncat>QUESTION: I see dogtail can work with GTK-enabled OpenOffice; is this the default in gutsy?
(12:26:36 PM) liw: the way we determine that a script worked is that it exits with a zero exit code -- this will hide some problems, but we'll worry about those at a later stage -- if a scipt crashes, it will return non-zero, so we'll notice that
(12:27:03 PM) liw: jimcooncat, I don't know if that is the default in gutsy, sorry
(12:27:15 PM) heno_: It is, yes
(12:27:26 PM) jimcooncat: thanks, this will be very useful
(12:27:27 PM) liw: but more generally speaking, AT-SPI / dogtail should be able to work with any gtk+ application, with few exceptions
(12:27:28 PM) heno_: I've run dogtail scripts on openoffice
(12:28:02 PM) liw: oh, and to see the exit code of a script, run this right after the script: echo $?
(12:28:04 PM) heno_: openoffice also works fairly well with Orca the screen reader
(12:28:16 PM) liw: $? is the shell's charming way to express "exit code of the previous command2
(12:28:20 PM) heno_: and that relies on AT-SPI
(12:28:33 PM) heno_: (sorry to interrupt)
(12:29:18 PM) liw: yeah, at-spi (and atk from -chat) is all part of the assistive technology stuff, and it's meant for screenreaders and such aids, and the way we're using it for testing is really an afterthought, but it works (really! it does!)
(12:29:58 PM) liw: ok, my qemu image upload finished, so you should be able to download http://people.ubuntu.com/~liw/autoguitest/autoguitest.qemu.lzma for a working hardy i386 basic gnome setup
(12:30:18 PM) liw: uncompress with unlzma, and run with: kvm -m 512 -hda autoguitest.qemu -snapshot
(12:31:05 PM) liw: the -m 512 is the amount of emulated memory, and -snapshot means that any changes you make won't be writte into the image (also meaning you will lose them once you quit kvm)
(12:31:12 PM) liw: but continuing to writing scripts
(12:31:20 PM) liw: er, not yet
(12:31:34 PM) liw: the username is "tomjon", the password is "password1" (the industry standard password!)
(12:31:41 PM) liw: and now, writing scripts
(12:32:23 PM) liw: the easy way of writing scripts is by using the dogtail-recorder program (part of python-dogtail) and use the program you want to write a test for with the keyboard only
(12:32:56 PM) liw: dogtail seems to be missing almost all mouse events, but gets keyboard events reliably, so it's best to stick to keyboarding
(12:33:31 PM) liw: so: start the program you want to write a script for, start dogtail-recorder, click on "record", do stuff, click on "stop", and then on "save", and you have a script
(12:33:41 PM) liw: this sounds too easy and wonderful to be true
(12:33:43 PM) liw: and it is
(12:33:55 PM) liw: the scripts pretty much always need some manual tweaking
(12:34:17 PM) liw: http://paste.ubuntu.com/4837/ has one example script
(12:34:24 PM) liw: straight from dogtail-recorder
(12:34:52 PM) liw: the first problem is that this script does not actually start the program, it assumes the program is already running
(12:35:29 PM) liw: we can fix this by replacing the first two "focus...." lines with "run('gcalctool')"
(12:35:50 PM) liw: http://paste.ubuntu.com/4836/ -- this is the script after editing -- there's more changes, I'll go through them next
(12:37:06 PM) liw: I find that it is best to write the scripts such that they are independent: they don't assume the program is already running, or anything else, except possibly that there aren't other programs running at the same tiem that might interfere (jimcooncat, hi), because those other programs are going to make things very complicated if we allow for them
(12:37:13 PM) liw: so, at least for now, we're keeping things simple
(12:37:44 PM) liw: the other problem with the original script in my pastebin is that I messed up while keyboarding and some silly and wrong stuff got recorded
(12:38:23 PM) liw: basically: I was driving the app with the keyboard, but making assumptions about the order of things in menus, which is dangerous, if we want to keep the scripts running from release to release
(12:38:33 PM) liw: and we want to be able to run the scripts from release to release, if possible
(12:38:45 PM) liw: less work for testers... :)
(12:39:00 PM) liw: so, I removed some useless keyboarding from the beginning of the script
(12:39:29 PM) liw: the final difference is that dogtail-recorder inserts focus.something stuff in various places, and it is meant to do that
(12:39:48 PM) liw: those focus things affect dogtails's idea of what has focus, not the window manager's
(12:40:26 PM) liw: unfortunately, when I run the scripts, the focus setting doesn't really work, and I'm not entirely sure why, but luckily it is OK to just remove them, at least as long as we stick to an application with one window (and no dialog windows)
(12:40:55 PM) liw: so the new version of the script is much shorter, and just toggles the various settings in the View menu of gcalctool
(12:41:05 PM) liw: any questions about this so far?
(12:41:55 PM) liw: those interested in danger and excitement may want to save http://paste.ubuntu.com/4836/ to foo.py and run it and see if it works
(12:42:06 PM) Grazieno: liw: it's ok for me
(12:42:15 PM) jimcooncat: So I can intersperse python code with the dogtail commands?
(12:42:38 PM) liw: oh, right: the dogtail scripts are pure Python code, which calls the dogtail Python library
(12:42:54 PM) liw: so if you know what you're doing, you can add any Python stuff in there that you wish
(12:43:35 PM) pranith: hello
(12:43:53 PM) pranith: is this where the bughelper session going to be held?
(12:44:16 PM) liw: getting scripts to work is sometimes a bit of a hassle: I haven't figured out a way to do it systematically yet, so that things always work the first time, but a little tweaking is usually all that is required
(12:44:39 PM) liw: it's best to keep scripts fairly small -- this means that when they break, or when they find a problem, it is easier to see what happens
(12:45:10 PM) liw: a 1000 line script that tests all of an application is harder to debug than a hundred 10-line scripts that each test a small part of the application
(12:46:03 PM) pixelpapst: pranith: you're interrupting a running session. join #ubuntu-classroom-chat and i'll fill you in
(12:46:18 PM) pranith: oops, sorry everyone
(12:46:48 PM) liw: if you write any scripts, we would like to ask you to upload them to https://wiki.ubuntu.com/Testing/Automation/Coverage and we can hopefully include them in the automatic desktop testing suite and eventually run them automatically as part of the Ubuntu release process
(12:47:29 PM) liw: if you can attach a small text file that explains what the tests are about (or include that in comments in the script itself), even better, but as long as you get scripts done, we're happy :)
(12:48:06 PM) liw: so, any questions now?
(12:48:31 PM) allinsane: can't get dogtail to run do they have an IRC?
(12:48:53 PM) liw: #dogtail on this IRC network, but I don't know how active that is
(12:49:13 PM) jimcooncat: Can dogtail read text from a widget back into a python variable?
(12:49:22 PM) Grazieno: <phoenix24_> QUESTION: Where can I get the schedule for the #ubuntu-classroom ?
(12:49:36 PM) liw: jimcooncat, in theory, yes, through the at-spi interfaces, but I'm not sure how to do that
(12:49:56 PM) liw: phoenix24_, https://wiki.ubuntu.com/UbuntuDeveloperWeek
(12:52:50 PM) liw: ok, I think we're about done, then ... ahead of time, even (my keyboard is smoking and my fingers are numb...)
(12:53:47 PM) jimcooncat: liw, what kinds of script would you like to see on your wiki page?
(12:54:15 PM) liw: jimcooncat, any script that verifies that some part of a GUI application in Ubuntu works
(12:54:37 PM) liw: jimcooncat, anything that interests you is good, and we're most interested in stuff that is included in the default installation
(12:55:45 PM) liw: those who had trouble getting dogtail to work: I am goign to have to leave in a few minutes, and I'm at fosdem over the weekend, but e-mail me at lars@canonical.com or ask me on irc on Monday, and I'll help you as best I can
(12:56:17 PM) liw: thank you, everyone
(12:56:30 PM) ***liw bows and exits gracefully
(12:56:34 PM) Grazieno: liw: very thanks!
(12:56:45 PM) jimcooncat left the room ("Ex-Chat").
(12:56:53 PM) barcc: liw: thanks