Xpresser

Differences between revisions 1 and 16 (spanning 15 versions)
Revision 1 as of 2010-05-18 14:15:03
Size: 2618
Editor: loquat
Comment:
Revision 16 as of 2012-09-20 23:59:09
Size: 6961
Editor: c-98-216-7-227
Comment:
Deletions are marked like this. Additions are marked like this.
Line 12: Line 12:
  * python-pyatspi
  * python-gtk2
Line 15: Line 13:
  * python-pyatspi2
  * python-cairo
  * python-gi-cairo

Xpresser was developed in Ubuntu Lucid (released in April of 2010), but is currently fully updated and available for the most recent Ubuntu releases. The project has an active team providing new features and updates on a regular basis.
Line 18: Line 22:
Xpresser is copyrighted by Canonical Ltd., and licensed under the [[http://www.gnu.org/licenses/lgpl.html|GNU LGPL]]. Xpresser is licensed under the [[http://www.gnu.org/licenses/lgpl.html|GNU LGPL]]
Line 20: Line 24:
== Downloading ==
Line 22: Line 25:
At the moment, Xpresser is available solely as a Bazaar branch in Launchpad. == Getting started ==
Line 24: Line 27:
Execute the following command to download it: === Downloading ===
Line 26: Line 29:
  * bzr branch lp:xpresser Both source and binary downloads are available through the project's PPA on Launchpad: https://launchpad.net/xpresser
Line 28: Line 31:
More details about the project may be obtained in the Launchpad page: === Installing ===
Installable deb packages can be found in the team PPA: https://launchpad.net/~xpresser-team/+archive/ppa
Installation from here is just an apt-get install away.
Line 30: Line 35:
  * https://launchpad.net/xpresser But you can also compile your own from the active code branch: https://code.launchpad.net/~xpresser-team/xpresser/trunk
Line 32: Line 37:
== Documentation == From source you will need to navigate the xpresser trunk directory to the setup.py script in its root. Run it.
Line 34: Line 39:
=== Getting started === Next step will be to import the function. From inside your branched directory run:
{{{
>>> from xpresser import Xpresser
}}}

=== Fire it up ===
Line 55: Line 65:
If you want to right-click instead, use the respective method:

{{{
>>> xp.right_click("my-button")
}}}

Besides using an image name as the argument, you can also use an image match instance as returned by find(), or an (x, y) pair.
Line 58: Line 75:
Instead of clicking, sometimes you may simply want to put the mouse cursor over something so that the interface reacts to the this event. This can be done in a similar way to clicking. Instead of clicking, sometimes you may simply want to put the mouse cursor over something so that the interface reacts to this event. This can be done in a similar way to clicking.
Line 66: Line 83:
Besides using an image name as the argument, you can also use an image match instance as returned by find(), or an (x, y) pair.
Line 68: Line 87:
You may not want to do any action on the image itself, but simply to wait until something is shown in the screen before doing an additional action. This can easily done as such: You may not want to do any action on the image itself, but simply to wait until something is shown in the screen before doing an additional action. This can easily be done as such:
Line 74: Line 93:
Besides using an image name as the argument, you can also use an image match instance as returned by find(), or an (x, y) pair.

=== Emulating the keyboard ===

If you want to use the keyboard to type some value in an edit box, for example, you can use the following method:

{{{
>>> xp.click("name")
>>> xp.type("John Doe")
>>> xp.type(["<F1>","I pressed the F1 key"])
}}}
Line 76: Line 107:
Sometimes, you may want to match an image even though it's slightly less similar to the original image you have taken the screenshot from, or you may want to click on a spot which is not quite the middle of the image you're matching. Sometimes, you may want to match an image even though it's slightly less similar to the original image you have taken the screenshot from, or you may want to click on a spot which is not quite the middle of the image you're matching.

Xpresser offers a convenient way to achieve these goals: just drop a file named {{{xpresser.ini}}} inside the images directory, and include sections in it resembling the following one:

{{{
[image edit]
filename = edit-text-box.png
similarity = 0.8
focus_delta = +50 +0
}}}

Each of these sections will customize the behavior for one image. The example above, for instance, will determine that the "edit" image will match even if it's only 80% similar to the original image, and if some action is performed on it (e.g. click), it will actually happen 50 pixels to the right side of the middle of the image matched.

=== Screencast ===
 * http://www.youtube.com/watch?v=b-xxBBPU9kU&feature=youtu.be

== Sample Test Case ==
=== Image Match ===
{{{
#
# Copyright (c) 2010 Canonical
#
# Written by Gustavo Niemeyer <gustavo@niemeyer.net>
#
# This file is part of the Xpresser GUI automation library.
#
# Xpresser is free software; you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License version 3,
# as published by the Free Software Foundation.
#
# Xpresser is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from xpresser.imagematch import ImageMatch, ImageMatchError
from xpresser.image import Image

from xpresser.lib.testing import TestCase


class ImageMatchTest(TestCase):

    def test_constructor(self):
        image = Image(width=40, height=50, focus_delta=(4, 5))
        match = ImageMatch(image, 100, 200, 0.5)
        self.assertEquals(match.image, image)
        self.assertEquals(match.x, 100)
        self.assertEquals(match.y, 200)
        self.assertEquals(match.similarity, 0.5)
        self.assertEquals(match.focus_point, (100+40//2+4, 200+50//2+5))

    def test_error_on_lack_of_height(self):
        image = Image(width=1)
        self.assertRaises(ImageMatchError, ImageMatch, image, 0, 0, 0)

    def test_error_on_lack_of_width(self):
        image = Image(height=1)
        self.assertRaises(ImageMatchError, ImageMatch, image, 0, 0, 0)
}}}

== Get Involved ==
=== Hit List ===
 1. '''Documentation''': Updating our wiki with new features and commands.
 1. '''Unit Tests''': We need unit [[https://code.launchpad.net/~xpresser-team/xpresser/trunk|tests]] for the existing features.
 1. '''New Features''': Check out our existing [[https://blueprints.launchpad.net/xpresser|Blueprints]] or create some yourself...then code!
 1. '''Test and Fix''': No project is perfect, log some [[https://bugs.launchpad.net/xpresser|bugs]], or fix [[https://code.launchpad.net/xpresser|some]].

What is Xpresser?

Xpresser is a Python module which enables trivial automation of Graphic User Interfaces (GUIs) via image matching algorithms.

Requirements

Xpresser depends on the following libraries to do its work:

  • python-opencv
  • python-numpy
  • python-pyatspi2
  • python-cairo
  • python-gi-cairo

Xpresser was developed in Ubuntu Lucid (released in April of 2010), but is currently fully updated and available for the most recent Ubuntu releases. The project has an active team providing new features and updates on a regular basis.

Xpresser is licensed under the GNU LGPL

Getting started

Downloading

Both source and binary downloads are available through the project's PPA on Launchpad: https://launchpad.net/xpresser

Installing

Installable deb packages can be found in the team PPA: https://launchpad.net/~xpresser-team/+archive/ppa Installation from here is just an apt-get install away.

But you can also compile your own from the active code branch: https://code.launchpad.net/~xpresser-team/xpresser/trunk

From source you will need to navigate the xpresser trunk directory to the setup.py script in its root. Run it.

Next step will be to import the function. From inside your branched directory run:

>>> from xpresser import Xpresser

Fire it up

Starting to use Xpresser is quite easy. All it needs to operate is a directory containing screen excerpts (images containing relevant crops taken out of one or more screenshots). These might be buttons, edit areas, image buttons, or anything else which could be relevant to perform an action on, or maybe simply to wait for its presence.

With this in place, an instance of the Xpresser class can be created:

>>> xp = Xpresser()
>>> xp.load_images("/path/to/images")

Clicking somewhere

If you want to press a button in the screen, for instance, you can take a screenshot of the button itself and maybe a bit of margin around it, and put the screenshot excerpt in a directory.

Then, assuming you've named the image my-button.png, you might simply do something like this:

>>> xp.click("my-button")

If you want to right-click instead, use the respective method:

>>> xp.right_click("my-button")

Besides using an image name as the argument, you can also use an image match instance as returned by find(), or an (x, y) pair.

Hovering over something

Instead of clicking, sometimes you may simply want to put the mouse cursor over something so that the interface reacts to this event. This can be done in a similar way to clicking.

Again, assuming you've named the image my-button.png, you might to do this:

>>> xp.hover("my-button")

Besides using an image name as the argument, you can also use an image match instance as returned by find(), or an (x, y) pair.

Waiting for things to show up

You may not want to do any action on the image itself, but simply to wait until something is shown in the screen before doing an additional action. This can easily be done as such:

>>> xp.wait("connection-established-indicator")

Besides using an image name as the argument, you can also use an image match instance as returned by find(), or an (x, y) pair.

Emulating the keyboard

If you want to use the keyboard to type some value in an edit box, for example, you can use the following method:

>>> xp.click("name")
>>> xp.type("John Doe")
>>> xp.type(["<F1>","I pressed the F1 key"])

Customizing the behavior on specific images

Sometimes, you may want to match an image even though it's slightly less similar to the original image you have taken the screenshot from, or you may want to click on a spot which is not quite the middle of the image you're matching.

Xpresser offers a convenient way to achieve these goals: just drop a file named xpresser.ini inside the images directory, and include sections in it resembling the following one:

[image edit]
filename = edit-text-box.png
similarity = 0.8
focus_delta = +50 +0

Each of these sections will customize the behavior for one image. The example above, for instance, will determine that the "edit" image will match even if it's only 80% similar to the original image, and if some action is performed on it (e.g. click), it will actually happen 50 pixels to the right side of the middle of the image matched.

Screencast

Sample Test Case

Image Match

#
# Copyright (c) 2010 Canonical
#
# Written by Gustavo Niemeyer <gustavo@niemeyer.net>
#
# This file is part of the Xpresser GUI automation library.
#
# Xpresser is free software; you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License version 3,
# as published by the Free Software Foundation.
#
# Xpresser is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
from xpresser.imagematch import ImageMatch, ImageMatchError
from xpresser.image import Image

from xpresser.lib.testing import TestCase


class ImageMatchTest(TestCase):

    def test_constructor(self):
        image = Image(width=40, height=50, focus_delta=(4, 5))
        match = ImageMatch(image, 100, 200, 0.5)
        self.assertEquals(match.image, image)
        self.assertEquals(match.x, 100)
        self.assertEquals(match.y, 200)
        self.assertEquals(match.similarity, 0.5)
        self.assertEquals(match.focus_point, (100+40//2+4, 200+50//2+5))

    def test_error_on_lack_of_height(self):
        image = Image(width=1)
        self.assertRaises(ImageMatchError, ImageMatch, image, 0, 0, 0)

    def test_error_on_lack_of_width(self):
        image = Image(height=1)
        self.assertRaises(ImageMatchError, ImageMatch, image, 0, 0, 0)

Get Involved

Hit List

  1. Documentation: Updating our wiki with new features and commands.

  2. Unit Tests: We need unit tests for the existing features.

  3. New Features: Check out our existing Blueprints or create some yourself...then code!

  4. Test and Fix: No project is perfect, log some bugs, or fix some.

Xpresser (last edited 2012-12-22 22:48:08 by c-24-147-75-163)