Ubuntu Wiki

   1 [18:31] <greyback> Hey all, I'm Gerry Boland, team lead of the Unity 2D project.
   2 [18:31] <greyback> For something like UbuntuTV to be a success, we need Unity2D to be rock solid!
   3 [18:31] <greyback> So I'm here to give a short 30 minute session about something almost as exciting: automated testing for Qt apps (and Unity2D)! :D
   4 [18:31] <greyback> Feel free to ask questions at any time, check the topic for instructions on how to do so.
   5 [18:32] <greyback> In Unity 2D, we have just introduced an automated User Experience test system, based on a test framework called "Testability Driver" (I'll just call it 'Testability' from now on).
   6 [18:32] <greyback> With this system, for every bux-fix and new feature, we:
   7 [18:32] <greyback> 1) automatically verify that all existing tests pass, so nothing is broken and
   8 [18:33] <greyback> 2) require that new tests are added to prevent regressions, thus in time developing a comprehensive test suite!
   9 [18:33] <greyback> I want to go into more detail about "Testability" and how everyone can use it.
  10 [18:33] <greyback> = What does Testability do? =
  11 [18:34] <greyback> Testability is for Qt-based applications (and Unity 2D is written in Qt, yay!).
  12 [18:34] <greyback> Its core feature is that it allows inspection of the tree of QObjects in a Qt application while it is running.
  13 [18:34] <greyback> It can read and write object properties, call methods and slots, verify signals are emitted, as well as faking mouse/keyboard/gesture inputs, grabbing visual outputs, and measure performance.
  14 [18:35] <greyback> And best of all, Testability is open source and maintained by Nokia!
  15 [18:35] <greyback> That means everyone can run and contribute tests! :)
  16 [18:36] <ClassBot> burli asked: is this built in Qt and language independend?
  17 [18:36] <greyback> burli: yes, it's build into Qt, from version 4.6 I think
  18 === Cantide is now known as CantideAFK
  19 [18:37] <greyback> It works with C++ and QML. If it uses Qt bindings then it will work too
  20 [18:37] <greyback> To show it off, here is a screengrab of the Testability Visualizer application which allows you to connect to a running application, dig into the QObject tree and investigate what's going on (here, the Unity 2D Launcher):
  21 [18:37] <greyback> http://i.imgur.com/psi7m.png
  22 [18:38] <greyback> On the left is an interactive snapshot of the UI, in the centre is the QObject tree which you can navigate, and on the right is the list of the selected QObject's properties, methods and signals you can interact with.
  23 [18:38] <greyback> On display in that screengrab are some of the attributes of the QDeclarativeImage object which draws the Terminal icon in the launcher (this is defined in QML!) - you can read off the source of the image, the x,y coordinates, width & height, and a whole lot more.
  24 [18:39] <greyback> All these properties, methods, signals, etc, are scriptable via Ruby. (I'll give a uber-simple example in a minute)
  25 [18:39] <greyback> Thus we can emulate every interaction that a user can make with the application, and determine the reaction.
  26 [18:39] <greyback> And all this power comes with almost no changes* to the source code!
  27 [18:39] <greyback> (* = just need to add object names, see later)
  28 [18:40] <greyback> This forms the foundation for the Unity 2D User Experience testing suite.
  29 [18:40] <greyback> = How Testability works =
  30 [18:41] <ClassBot> TwoD asked: Does Testability include tests for asserting the number of keystrokes/clicks required for completing user task "X" remains sane?
  31 [18:42] <greyback> TwoD: you write a test, by providing every click and keystroke, and check the behaviour matches what you expect.
  32 [18:42] <greyback> TwoD: you control the number of clicks
  33 [18:42] <greyback> Testability works as follows (have a look at this image too, it might help http://tinyurl.com/7dj5jdw):
  34 [18:42] <greyback> - *any* Qt application using Qt4.6+ can be tested by executing it with the "-testability" switch.
  35 [18:43] <greyback> - a standalone server "qttasserver" runs in the background.
  36 [18:43] <greyback> - With the -testability switch, the Qt library tries to load a "libtestability.so" plugin which establishes a connection between the application and qttasserver, giving qttasserver access to the root node of the QObject tree.
  37 [18:43] <greyback> - qttasserver then climbs the QObject tree, reads all the info it can and converts it to an XML format. It also can receive commands and cause the application to react upon them (click here, type, etc..).
  38 [18:44] <greyback> - A series of Ruby scripts connect to qttasserver, receive and parse this XML and allow us to script tests and interactions with the application.
  39 === fenris_ is now known as Guest88852
  40 [18:45] <greyback> Then we use Ruby's test runner systems (Test::Unit or Cucumber) to write and run tests
  41 [18:45] <greyback> [The Unity 2D test framework also includes an extra helper library to control the X server, to control the mouse and keyboard, and to manipulate windows on the desktop. As far as X is concerned, a human is controlling it.]
  42 [18:45] <greyback> With the ability to fake any form of user input, and directly read the output from the shell applications themselves, we can test almost every behaviour of the application. And as a result, the quality of the experience will only go up!
  43 === fenris__ is now known as Guest62166
  44 [18:46] <ClassBot> calmpitbull asked: What is the best way to get started with programing unity 2d, lenses and scopes
  45 [18:46] <greyback> calmpitbull: there's a session on Unity and lenses shortly. Lenses are compatible with both Unity and Unity2D
  46 [18:47] <greyback> calmpitbull: if youdd like to contribute to Unity 2D, please ping me after this presentation. We'd be delighted to have you!
  47 [18:47] <ClassBot> ashickur-noor asked: What is the best language to put on the Back-end of QT
  48 [18:48] <greyback> ashickur-noor: (you've gone?) Well I'm a fan of C++ and QML. QML makes UI design and creation really easy
  49 [18:48] <ClassBot> pawel_st asked: ​ Testability seems to be well suited for scenarios where object attributes can be tested... Is there any way you can test if drawing errors etc. occur?
  50 [18:49] <greyback> pawel_st: Testabiity has access to the QPainter, so whatever Qt thinks is being drawn to screen, Testability can grab. It can then perform visual comparison on images, to check for errors
  51 [18:49] <greyback> = How to get & use Testability =
  52 [18:49] <greyback> Unfortunately Testability is not available in Ubuntu's repos just yet, but I have it packaged in a PPA.
  53 [18:49] <greyback> Installation takes a few steps, so I suggest that interested people consult this wiki page: https://wiki.ubuntu.com/Unity2DTestability
  54 [18:50] <greyback> Tests are just Ruby scripts, so running tests is just a matter of running a Ruby script!
  55 [18:50] <greyback> I think it easiest to show how to use Testability with an example ruby script: http://pastebin.com/AWWYYPN8
  56 [18:50] <ClassBot> There are 10 minutes remaining in the current session.
  57 [18:51] <greyback> That test checks
  58 [18:51] <greyback> 1. the Launcher is 65 pixels wide
  59 [18:51] <greyback> 2. that the Terminal tile has the correct icon
  60 [18:51] <greyback> There is some boiler-plate code there, but the bits I want to point out are:
  61 [18:51] <greyback> @app.LauncherView()['width']
  62 [18:51] <greyback> This causes Testability to search for an object of type "LauncherView()" in the QObject tree, and if found reads its "width" property and returns it (throws exception if not).
  63 [18:51] <greyback> Then in Ruby, we can check this value matches what we expect (65).
  64 [18:52] <greyback> The second test does a similar operation, but needs a little more help.
  65 [18:52] <greyback> As there are many tiles, we need to help Testability to find the exact tile we want. We do this by adding some object names ("main", "Terminal",..) to the C++/QML.
  66 [18:52] <greyback> By consulting with http://i.imgur.com/psi7m.png, maybe you can see how Testability navigates the tree to find the icon source for the Terminal tile!
  67 [18:52] <greyback> Once you can track down objects uniquely, you can then start interacting with them, send mouse clicks, set properties, call methods, etc.
  68 [18:52] <greyback> This power gives you great flexibility in testing your application!
  69 [18:53] <greyback> = Demo & conclusion =
  70 [18:53] <greyback> As a demo, here is a video of a part of the Unity 2D test suite in action. On the right is Ubuntu Oneiric running inside VirtualBox, where the Launcher will be tested. On the left is my host machine terminal running the test suite.
  71 [18:53] <greyback> https://wiki.ubuntu.com/Unity2DTestability?action=AttachFile&do=get&target=testability.ogv
  72 [18:53] <greyback> Various hide/show tests are being performed, with windows in the way, mouse being controlled, keyboard shortcuts being pressed etc.
  73 [18:54] <greyback> There's something very strange about watching a desktop doing things when nobody is controlling it!! :)
  74 [18:54] <ClassBot> Atlantic777 asked: any documentation on how to write good test cases?
  75 [18:54] <greyback> Atlantic777: there is some at Nokia's website on Testability(https://projects.developer.nokia.com/Testabilitydriver/wiki), and I plan to write my own shortly.
  76 [18:55] <ClassBot> Saviq asked: there seems to be a split between the application and the tester, can this work across the network?
  77 [18:55] <greyback> Saviq: yes. In that video above, all the communication between the application and the test machine is via TCP/IP
  78 [18:56] <ClassBot> There are 5 minutes remaining in the current session.
  79 [18:56] <ClassBot> Ceno asked: I see there's no support for 10.04 in the PPA.  Can this be arranged, or since the default 10.04 qt version is bellow 4.6 it doesn't even matter?
  80 [18:56] <greyback> Ceno: it's my PPA, so if you need it, I can try. Version less than 4.6 can be tested too, they need a few extra lines of code in the application only.
  81 [18:57] <ClassBot> bjhaid asked: can you please post link to sample test
  82 [18:57] <greyback> bjhaid: http://pastebin.com/AWWYYPN8 for super-simple test
  83 [18:58] <greyback> bjhaid: otherwise, check http://bazaar.launchpad.net/~unity-2d-team/unity-2d/trunk/files/head:/tests/
  84 [18:58] <greyback> Our policy is that every new feature and bug fix in Unity 2D will have a test attached, to ensure no regressions.
  85 [18:58] <greyback> This will ensure Unity 2D remains reliable, consistent and fast. Thanks to Testability!
  86 [18:58] <greyback> Thanks for your attention everybody!
  87 [18:59] <greyback> I've a minute for questions
  88 [18:59] <ClassBot> pawel_st asked: ​ Can it run headless, e.g. for automated integration testing?
  89 [18:59] <greyback> pawel_st: yes. It should run in Xvfb
  90 [18:59] <ClassBot> bjhaid asked: any gem required to run the test or it runs on standard ruby modules?
  91 [19:00] <greyback> bjhaid: just "gem require" is needed for some reason I'm unsure of
  92 [19:00] <greyback> Ok I'm done. Thanks everyone