Ubuntu Wiki

   1 [21:01] <barry> welcome to "rock solid python development with unittest/doctest".  today i'm going to give a brief introduction to unit- and doc- testing your python applications, and hooking these into the debian packaging infrastructure.
   2 [21:01] <barry> raise your hand if you're already unashamedly obsessed with testing :)
   3 === ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu App  Developer Week - Current Session: Rock solid Python development with unittest/doctest - Instructors: barry
   4 [21:02] <barry> since it would take way more than one hour, i'm not going to give a deep background on testing, or the python testing culture, or python testing tools.  there are a ton of references out there.  two resources i'll give right up front are the testing-in-python mailing list <http://tinyurl.com/2bl2gk> and the python testing tools taxonomy <http://tinyurl.com/msya4>.  python has a *very* rich testing culture, and i encourage you to explore
   5 [21:02] <barry> it!
   6 [21:02] <ClassBot> Logs for this session will be available at http://irclogs.ubuntu.com/2011/04/11/%23ubuntu-classroom.html following the conclusion of the session.
   7 [21:03] <barry> sorry, i'm going to start over because of the classbot delay...
   8 [21:03] <barry> welcome to "rock solid python development with unittest/doctest".  today i'm going to give a brief introduction to unit- and doc- testing your python applications, and hooking these into the debian packaging infrastructure.
   9 [21:03] <barry> raise your hand if you're already unashamedly obsessed with testing :)
  10 [21:03] <barry> since it would take way more than one hour, i'm not going to give a deep background on testing, or the python testing culture, or python testing tools.  there are a ton of references out there.  two resources i'll give right up front are the testing-in-python mailing list <http://tinyurl.com/2bl2gk> and the python testing tools taxonomy <http://tinyurl.com/msya4>.  python has a *very* rich testing culture, and i encourage you to explore
  11 [21:03] <barry> it!
  12 [21:03] <barry> there's a lot you can do right out of the box, and that's where we'll start.  michael foord is hopefully here today too; he's the author of unittest2, a standalone version of all the wizzy new unittest stuff in python2.7
  13 [21:03] <barry> for now, we'll keep things pretty simple, and the examples should run in python2.6 or python2.7 with nothing extra needed.
  14 [21:03] <barry> for those of you with bazaar, the example branch can be downloaded with this command: bzr branch lp:~barry/+junk/adw
  15 [21:04] <barry> if you can't check out the branch, you can view it online here:
  16 [21:04] <barry> http://bazaar.launchpad.net/~barry/+junk/adw/files
  17 [21:04] <barry> i'll pause for a few moments so that you can grab the branch or open up your browser
  18 [21:05] <barry> here's a quick overview of what we'll be looking at: a quick intro to unittesting, a quick intro to doctesting, hooking them together in a setup.py, hooking them into your debian packaging.
  19 [21:06] <barry> let's first look at a simple unittest.  if you've downloaded the branch referenced above, you should open the file adw/tests/test_adding.py in your editor.
  20 [21:06] <barry> http://bazaar.launchpad.net/~barry/+junk/adw/view/head:/adw/tests/test_adding.py
  21 [21:06] <barry> the adw package is really stupid.  it has one function which adds two integers together and returns the results.  i won't talk much about test driven development (tdd) here, but i highly encourage you to read up on that and to practice tdd diligently!  these tests were developed using tdd.
  22 [21:07] <barry> anyway, looking at test_adding.py, you can see one test class, called TestAdding.  there are some other boilerplate stuff in test_adding.py that you can mostly ignore.  the TestAdding class has one method, test_add_two_numbers().  this method is a unittest.  you'll notice that it calls the add_two_numbers() function (called the "system under test" or sut), and asserts that the return value is equal to 20.  pretty simple.
  23 [21:07] <barry> look below that at the test_suite() function.  this is mostly boilerplate used to get your unittest to run.  the function creates a TestSuite object and adds the TestAdding class to it.  python's unittest infrastructure will automatically run all test_*() methods in the test classes in your suite.
  24 [21:08] <barry> let's run the tests.  type this at your shell prompt:
  25 [21:08] <barry> $ python setup.py test
  26 [21:08] <barry> (without the $ of course)
  27 [21:08] <barry> the first time you do this, your package will get built, then you'll see a little bit of verbose output you can ignore, and finally you'll see that two tests were run.  ignore the README.txt doctest for the moment.
  28 [21:09] <barry> if you want to see what a failing test looks like, uncomment the test_add_two_numbers_FAIL() method and run `python setup.py test` again.  be sure to comment that back out afterward though! :)
  29 [21:09] <barry> everybody with me so far?  i'll pause for a few minutes to see if there are any questions up to now
  30 [21:10] <barry> !q
  31 [21:11] <barry> so, obviously the more complicated your library is, the more test methods, test classes, and test_*.py files you'll have.  i probably won't have time to talk about test coverage much, but there are excellent tools for reporting on how much of your code is covered by tests.  you obviously want to aim for 100% coverage.
  32 [21:11] <barry> okay, let's switch gears and look at a doctest now.  go ahead and open adw/docs/README.txt
  33 [21:11] <barry> http://bazaar.launchpad.net/~barry/+junk/adw/view/head:/adw/docs/README.txt
  34 [21:12] <barry> doctests are *testable documentation*.  the emphasis is on the documentation aspects of these files, and in fact there are excellent resources for turning doctests into actual documentation, e.g. http://packages.python.org/flufl.i18n/
  35 [21:12] <barry> doctests are written using restructured text, which is a very nice plain text human readable format.  the key thing for testing is to notice the last three lines of the file.  see the two lines that start with >>>
  36 [21:12] <barry> (aside: sphinx is the tool to turn rest documention into html, pdf, etc.)
  37 [21:13] <barry> and it's very well integrated with setup.py and the python package infrastructure
  38 [21:13] <barry> that's a python interpreter prompt, and doctests work by executing all the code in your file that start with the prompt.  any code that returns or prints some output is compared with text that follows the prompt.  if the text is equivalent, then the test passes, otherwise it fails.
  39 [21:14] <barry> oh, i should mention.  "doctests" can mean one of two things.  you can actually have testable sections in your docstrings, or you can have separate file doctests.  by personal preference, i always use the latter
  40 [21:16] <barry> btw, the use of doctests is somewhat controversial in the python world.  i personally love them, others hate them, but i think everyone agrees they do not replace unittests, but i think they are an excellent complement.  anyway, if we have time at the end we can debate that :)
  41 [21:16] <barry> !q
  42 [21:17] <barry> in this example, add_two_numbers() is called with two integers, and it returns the sum.  if you were to type the very same code at the python interpreter, you'd get 12 returned too.  doctest knows this and compares the output
  43 [21:17] <barry> run `python setup.py test` again and look carefully at the output.  you'll see that the README.txt doctest was run and passed.  if you change that 12 to a 13, you'll see what a failure looks like (be sure to change it back afterward!)
  44 [21:17] <barry> i'll pause for a few minutes to let folks catch up
  45 [21:18] <ClassBot> RawChid asked: So doctest is one way to do uittesting in Python?
  46 [21:19] <barry> RawChid: i'd say one way to do *testing*, which i'm personally a big fan of.  i love writing documentation first because it ensures that i can explain what i'm doing.  if i can't explain it, i probably don't understand it.  but for really thorough testing, you must add unittests.  e.g. you typically do not want to do corner case and error cases in doctests.
  47 === JasonO_ is now known as JasonO
  48 [21:20] <barry> although the heretic in me says you should try :)
  49 [21:20] <barry> unfortunately, python's unittest framework does not run doctests automatically.  if you look in the adw/tests directory, you'll see a file called test_documentation.py.  you don't need to study this much right now, and you are free to copy this into your own projects.  it's just a little boiler plate to hook up doctests with the rest of your test suite.  it looks for files inside docs/ directories that end in .txt or .rst (the reST
  50 [21:20] <barry> standard extension) and adds them to the test suite.  once you have test_documentation.py, you never need to touch it.  just add more .txt and .rst files to your docs directories, and it will work automatically.
  51 [21:21] <barry> http://bazaar.launchpad.net/~barry/+junk/adw/view/head:/adw/tests/test_documentation.py
  52 [21:21] <ClassBot> tronda asked: Is doctest somewhat similar to the BDD movement?
  53 [21:21] <barry> tronda: probably related.  i don't know too much of the details but i think there are better tools for doing bdd in python
  54 [21:21] <barry> voidspace might know more about that
  55 [21:22] <barry> everybody with me so far?
  56 [21:22] <barry> time to switch gears a little.  how do you hook up your unittests and doctests to your setup.py file so that you also can run `python setup.py test`?  open up setup.py in your editor and we'll take a look
  57 [21:23] <barry> http://bazaar.launchpad.net/~barry/+junk/adw/view/head:/setup.py
  58 [21:23] <barry> notice first that setup.py imports distribute_setup and then makes a function call.  you'll see the file distribute_setup.py in the branch's top level directory.  this means my package uses *distribute* which is the successor to setuptools.  i highly recommend it, but if you don't know what that is, you can just cargo cult this bit.
  59 [21:23] <barry> anyway, the setup.py is fairly simple.  you'll just notice one thing in the second to last line.  it sets `test_suite` to `adw.tests`.  the value is a python package path and it references the directory adw/tests.  this is how you hook up your test suite to setup.py.  when you run `python setup.py test` it looks at this test_suite key, and runs all the files that look like test_*.py
  60 [21:24] <barry> voidspace mentions in #u-c-c that we're pretty sure this is a setuptools extension to the standard distutils setup.py.  so it'll probably work for either setuptools or distribute
  61 [21:25] <barry> we have two of those of course!  test_adding.py and test_documentation.py, and the testing infrastructure automatically finds these, and makes the appropriate calls to find what tests to run.  so that little test_suite='adw.tests' line is all you need to hook your tests into setup.py
  62 [21:25] <barry> so far so good.  now let's look at how to hook your python test suite into your debian packaging so that your tests always run when your package is built.  if you're not into debian packaging you can ignore the next couple of minutes.
  63 [21:26] <barry> open up debian/rules in your editor.
  64 [21:26] <barry> http://bazaar.launchpad.net/~barry/+junk/adw/view/head:/debian/rules
  65 [21:26] <barry> first thing to notice is that my package uses dh_python2, which is the new goodness replacing python-central and python-support.  i highly recommend it as it can make your debian packaging of python code really really easy.  you can see there's not much to my rules file.
  66 [21:27] <barry> i won't go into dh_python2 much right now, but you can look at the debian wiki for more details http://wiki.debian.org/Python
  67 [21:27] <barry> for today's class, there are really three parts to take a look at.  the first is the PYTHON2 line.  what this does is ensure that your tests will be run against all supported versions of python (2.x) on your system, not just python2.6 or python2.7.  the commented out line for PYTHON3 will do something similar for python3
  68 [21:27] <barry> (e.g. line 3)
  69 [21:27] <barry> remember that ubuntu 11.04 supports both python2.6 and 2.7
  70 [21:27] <barry> aside: it is possible to write your package for both python2 and python3, and to run all the tests into both.  we won't have time to talk about that today though.
  71 [21:27] <barry> so the next thing to look at is the line that starts `test-python%`.  this is a wildcard rule that is used to run the setup.py test suite with every supported version of python2 on your system.  you'll notice the -vv which just increases the verbosity.
  72 [21:28] <barry> (e.g. line 9)
  73 [21:28] <barry> override_dh_auto_test line then expands the PYTHON2 variable to include all the supported versions of python2, and it runs the test-python% wildcard rule for each of these.  thus this hooks in the setup.py tests for all versions of python2.  the override is currently needed because dh_auto_test doesn't know about `python setup.py test` yet.
  74 [21:28] <barry> i won't go into the specifics of packaging building right now, but i've done a build locally, and the results are available here: http://pastebin.ubuntu.com/592711/
  75 [21:28] <ClassBot> eolo999 asked: I'm very comfortable with nosetests; is there a particular reason why you left it out from the session?
  76 [21:28] <barry> scroll down to line 251 and you'll see the test suite getting run for python2.7.  scroll down to line 268 and you'll see it getting run for python2.6.  the nice thing about this is that if you get a failure in either test suite, your package build will also fail.  this is a great way to ensure really high quality (i.e. rock solid :) python applications in ubuntu.
  77 [21:29] <ClassBot> jderose asked: I got the impression that setuptools wasn't well maintained lately, wasn't regarded as the way forward, esp with Python3 - is that true, WWBWD? :)
  78 [21:30] <barry> eolo999: mostly just to keep things simple.  voidspace in #u-c-c says that the main advantage of nosetests is the test runner, so it you can basically use all the techniques here with nose
  79 [21:30] <barry> i'm pretty sure that the future plans voidspace has for unittest2 include integrating nose more as a plugin than as a separate tool
  80 [21:31] <barry> btw, that's about all the canned stuff i have prepared, so i welcome questions from here on out
  81 [21:31] <barry> just ask them in #ubuntu-classroom-chat and we'll post the answers here
  82 [21:32] <barry> jderose: i'd say that's correct, though setuptools does get occasional new releases.  distribute is the maintained successor to setuptools, but for python3 distutils2 will be the way forward
  83 [21:32] <barry> i admit that it's all very confusing :)
  84 [21:33] <barry> but my recommendation would be: use distribute for python2 stuff, and for python3 stuff if you want the same code base to be compatible with 2.x and 3.x (i.e. via 2to3).  this is a great way to support both versions of python
  85 [21:34] <barry> oh yes, distutils2 will be called 'packaging' in python 3.3 and it will come in the stdlib
  86 [21:35] <barry> from #u-c-c:
  87 [21:35] <barry> <voidspace> barry is correct, I have plans for unittest to become more
  88 [21:35] <barry>             extensible (plugins) that should allow nose to become much simpler
  89 [21:35] <barry>             and be implemented as plugins for unittest
  90 [21:35] <barry> <voidspace> at the moment nose is convoluted and painful to maintain because
  91 [21:35] <barry>             unittest itself is not easy to extend
  92 [21:35] <barry>  
  93 [21:35] <barry> also lvh mentions trial, which is twisted's test runner.  for mailman3 i use zc.testing which is zope's test runner
  94 [21:36] <barry> so yeah, there are lots of testing tools out there :)
  95 [21:36] <barry> !q
  96 [21:37] <barry> <jderose> QUESTION: so is it okay/encouraged to run your python tests in PPA
  97 [21:37] <barry>           builds, say for daily recipes and whatnot?
  98 [21:38] <barry> jderose: i don't recall a discussion about it one way or the other.  personally, i would enable tests for all package builds, just to ensure that what you deploy has not regressed.
  99 [21:38] <barry> however, you do need to be careful that your test suite can *run* in a ppa environment
 100 [21:39] <barry> this may not always be the case.  some test suites require resources that are not available on the buildds.  those tests would obviously cause problems when your ppa were built
 101 [21:40] <barry> in those cases, it may be best to have more than one "layer" of tests.  one that gives good coverage and high confidence against regressions, but requires no expensive or external resources.  and a full test suite you can run locally with whatever crazy stuff you need
 102 [21:40] <barry> mocks might be a good answer to help with that
 103 [21:40] <barry> qwebirc57920: ppa == personal package archive
 104 [21:41] <barry> https://help.launchpad.net/Packaging/PPA
 105 [21:41] <barry> QUESTION: How do I know what resources are available on the
 106 [21:41] <barry>              buildds?
 107 [21:41] <barry> yeah, good question :)  ask on #launchpad or launchpad-dev, or just try it and see what fails ;)  there should be better documentation about that on help.l.net
 108 [21:42] <barry> <jderose> QUESTION: so if `test` requires a lot more dependencies than
 109 [21:42] <barry>           `install`, should we just put those all in Build-Depends?  when will
 110 [21:42] <barry>           we get Test-Depends?  :)
 111 [21:42] <barry> jderose: excellent question.  for now, i recommend build-depends
 112 [21:43] <barry> <tronda> Question: In the Java space there's a lot of mocking
 113 [21:43] <barry>          tools/libraries. Any need for that in Python - if so - which are the
 114 [21:43] <barry>          recommended ones?
 115 [21:43] <barry> voidspace can tell you how many mock libraries are available in python!  answer is *lots*
 116 [21:44] <barry> btw, please note that there are tools (such as pkgme and stdeb) that can debianize your setup.py based python project.  they do a pretty good job, though i'm not sure they turn test-requires into build-depends.
 117 [21:45] <barry> <jderose> QUESTION: you mentioned "layering" tests into light/heavy - what a
 118 [21:45] <barry>           good way of doing that?
 119 [21:45] <barry>  
 120 [21:47] <barry> jderose: i think this depends on the test runner you use.  python's stdlib for example uses -u flag to specify additional resources to enable (e.g. largefile).  most test runners have some way of specifying a subset of all tests to run and what i would do is in your debian/rules file, file the right arguments to your test runner to run the tests you can or want to run
 121 [21:47] <barry> note that in my debian/rules file, i set it up to run 'python setup.py test -vv' but really, it can run any command with any set of options
 122 [21:47] <barry> <chadadavis> QUESTION: to different doc tests share a common environment /
 123 [21:47] <barry>              namespace? Can I make them explicitly separate / explicitly
 124 [21:47] <barry>              shared?
 125 [21:47] <barry>  
 126 [21:48] <barry> chadadavis: all the doctests in a single file or docstring share the same namespace.  one of the criticisms of doctests is that it builds up state as it goes so it can sometimes be difficult if a test later in the file fails, to determine what earlier state caused the failure.
 127 [21:48] <barry> i think that just means you have to be careful, and also, keep your doctests focussed
 128 [21:49] <barry> not too big
 129 [21:49] <barry> you really just have to understand when and where each tool (unittest or doctest) is appropriate
 130 [21:50] <barry> voidspace also points out that every line in a doctest gets executed, even if there are failures (though i *think* there's a fail to cause it to bail on the first failure)
 131 [21:50] <barry> i'll just say that that can be an advantage or disadvantage depending on what you like and what you're trying to do :)
 132 [21:51] <barry> looks like we have a few minutes left.  are there any other questions?
 133 [21:52] <ClassBot> There are 10 minutes remaining in the current session.
 134 [21:52] <akgraner> <jderose> QUESTION - what's the status of 3to2?  write Python3 is so wonderful, i'd rather go that way than 2to3
 135 [21:52] <barry> i'll just say again what an excellent resource the testing-in-python mailing list is.  i highly recommend you join!
 136 [21:53] <barry> voidspace answers this as well as i could:
 137 [21:53] <barry> <voidspace> jderose: packaging (distutils2) is now using 3to2 rather than 2to3
 138 [21:53] <barry> <voidspace> jderose: so although I've not used it myself, it must be in a
 139 [21:53] <barry>             pretty good state  [16:53]
 140 [21:53] <barry>  
 141 [21:53] <barry> i've also not used 3to2 myself
 142 [21:53] <barry> much
 143 [21:54] <barry> fwiw, if you look at my test_documentation.py file, you'll see how you can do setups and teardowns for doctests
 144 [21:54] <barry> it also does fun stuff like set __future__ flags for the doctest namespace
 145 [21:56] <barry> voidspace says in #u-c-c that sphinx has support for doctests through its doctest:: directive
 146 [21:57] <ClassBot> There are 5 minutes remaining in the current session.
 147 [21:57] <barry> well, time is almost up, so let me thank you all for attending!  i know there was a lot of material and i blew through it pretty fast
 148 [21:57] <barry> in closing, i'll say that while we can all debate this or that detail of testing, there's no debate that testing is awesome and we all should do more of it!
 149 [21:58] <barry> big thanks to my colleague voidspace for helping out!