Ubuntu Wiki

   1 [20:00] <barry> we'll start in just a minute or two
   2 [20:01] <ClassBot> Logs for this session will be available at http://irclogs.ubuntu.com/2011/07/11/%23ubuntu-classroom.html following the conclusion of the session.
   3 [20:06] <barry> hello everyone and welcome to my session.  today we're going to talk about packaging python libraries using dh_python2 and dh_python3
   4 [20:06] <barry> please feel free to ask questions at any time over in #ubuntu-classroom-chat
   5 [20:06] <barry> so first some background:
   6 [20:06] <barry> there are two common legacy python helpers you'll find in various packages, python-central and python-support
   7 [20:07] <barry> python-central has been deprecated for a while, and python-support was just recently deprecated.
   8 [20:08] <barry> of course, many packages have still not been converted, and we had a jam session a week or so back where folks from the community helped convert packages on the ubuntu cds.  if we have time and interest, i can talk more about those transitions
   9 [20:08] <barry> today we have the new goodness for packaging python2 stuff: dh_python2
  10 [20:09] <barry> the only helper for packaging python3 stuff is: dh_python3
  11 [20:10] <barry> using dh_python2 can make most of your packaging work almost too trivial.  many many packages can have no more than a 3 line rules file
  12 [20:10] <barry> provided you already have a good setup.py
  13 [20:10] <ClassBot> mhall119 asked: can you use dh_python2 for packaging on Lucid yet?
  14 [20:11] <barry> mhall119: we are working on a back port of the full toolchain for lucid.  many people need this, so stay tuned!
  15 [20:12] <barry> let's look at a simple package to see what this would look like.  does everybody know how to use bzr to grab branches from launchpad?
  16 [20:13] <barry> okay, here's the url to the basic, un-debian-packaged version: lp:~barry/+junk/stupid
  17 [20:13] <barry> bzr branch lp:~barry/+junk/stupid
  18 [20:14] <barry> now, this is a very simple python package, but it does have one interesting thing: it has an extension module
  19 [20:14] <barry> if you look in src, you'll see the C file containing the extension module
  20 [20:14] <barry> if you look in stupid, you'll find the python code that wraps that
  21 [20:14] <barry> notice the unit tests :)
  22 [20:15] <barry> you can take a look at the setup.py to see it's a fairly typical thing
  23 [20:15] <barry> it's got an ext_modules defined for the extension, a few other bits of metadata, and it identifies the header file
  24 [20:15] <barry> note too, the test_suite key which names the unit tests
  25 [20:16] <barry> you could install this into a virtualenv using either python2.6, 2.7, or 3.2 using the following commands:
  26 [20:16] <barry> (this should work on natty)
  27 [20:16] <barry> virtualenv -p python2.7
  28 [20:17] <barry> oops
  29 [20:17] <barry> virtualenv -p python2.7 /tmp/27
  30 [20:17] <barry> source /tmp/27/bin/activate
  31 [20:17] <barry> python setup.py install
  32 [20:18] <barry> python -c 'import stupid; stupid.yes()'
  33 [20:18] <barry> that should print 'yes'
  34 [20:18] <barry> then run `deactivate` to get out of the virtualenv
  35 [20:19] <barry> you could substitute the following for the -p option python2.6 or python3.2 and that would give you a virtualenv with the appropriate python
  36 [20:19] <ClassBot> NMinker asked: What is virtualenv? Is that a Virtual Environment?
  37 [20:20] <barry> ah.  virtualenv is a python tool for creating isolated development environments.  with a virtualenv, you can install stuff locally for testing without affecting your system python
  38 [20:20] <barry> it's a *very* handy tool if you're working on python code
  39 [20:20] <barry> sudo apt-get install python-virtualenv
  40 [20:21] <barry> okay, so "stupid" is a simple python package with a good setup.py, and which is compatible with python 2.6, 2.7 and 3.2
  41 [20:21] <barry> how do we turn that into a debian package?
  42 [20:22] <barry> first we have to create the debian source package from the python package, then we can upload that source package to a ppa, or build it locally with pbuilder or sbuild
  43 [20:22] <barry> i am kind of assuming folks know basic packaging stuff, like how to use pbuilder, debuild, and such..
  44 [20:23] <barry> okay, so let's create the source package
  45 [20:23] <barry> first, i'll introduce you to a very nice, new tool which can almost always get you started quickly.
  46 [20:24] <barry> https://launchpad.net/pkgme
  47 [20:24] <barry> pkgme is actually a packaging framework.  it's not tied specifically to python, although it is written in python, and supports packaging python things
  48 [20:25] <barry> it knows about other languages and such, but for our purposes, it does a great job of creating the initial debian/ directory layout based on your setup.py
  49 [20:25] <barry> i highly recommend grabbing the ppa, using these commands:
  50 [20:25] <barry> sudo add-apt-repository ppa:pkgme-committers/dev
  51 [20:25] <barry> sudo apt-get update
  52 [20:25] <barry> sudo apt-get install pkgme
  53 [20:26] <barry> once it's installed, you just run this commands from the directory containing your setup.py
  54 [20:26] <barry> pkgme
  55 [20:26] <barry> :)
  56 [20:26] <barry> that's it
  57 [20:26] <barry> now, i've done this for you, so if you don't want to install the ppa
  58 [20:26] <barry> you can just do this:
  59 [20:26] <barry> bzr branch lp:~barry/+junk/stupid.pkgme
  60 [20:27] <barry> why don't you run pkgme locally, or grab the branch.  i'll give you a minute or so and then we'll look at the details
  61 [20:27] <barry> pkgme knows about dh_python2 so it does the right thing
  62 [20:28] <barry> notice that you've now got a minimal debian/ directory.  yay!  you have a source branch
  63 [20:28] <barry> or "packaging branch"
  64 [20:29] <barry> take a look at the debian/control file.  if you have any packaging experience, you'll see this one is bare minimum, but adequate to start with
  65 [20:30] <barry> the important things to note here are that it has a proper Build-Depends: line, and it's grabbed a few meta bits from the setup.py.  it's missing a description (that is because the setup.py doesn't have one, not because pkgme missed it), so you'd want to fill that out
  66 [20:30] <ClassBot> NMinker asked: What's the difference using pkgme and dh_make to create the debian folder?
  67 [20:31] <barry> pkgme is a framework where rules can be added for more specific knowledge of particular languages, classes of packages, etc.  you could use either tool, but i like where pkgme is going, and it has very good python rules
  68 [20:31] <barry> now, bring up debian/rules in your editor, because this is where the fun stuff happens
  69 [20:32] <barry> you can see, this is just a 3 line rules file essentially, and i'll step through what is happening
  70 [20:32] <barry> the first line isn't interesting, it's just standard debian packaging
  71 [20:33] <barry> ah, slight detour
  72 [20:33] <barry> jykae: noticed that <Python.h> could not be found, and here's why
  73 [20:34] <barry> pkgme actually didn't quite do the right thing with the Build-Depends line (yes, i will file a bug :)
  74 [20:34] <barry> it added a dependency on python-all, but because stupid has an extension module, it needs to be compiled by the c compiler.  thus it needs the python-all-dev package, which includes python's own header files and such
  75 [20:35] <barry> so you would need to change the Build-Depends line to be python-all-dev
  76 [20:35] <barry> anyway, back to the rules file
  77 [20:35] <barry> the %: line is fairly standard stuff, and introduces the make target
  78 [20:35] <barry> it basically matches anything
  79 [20:35] <barry> the really fun stuff is in the next line
  80 [20:36] <barry> dh is the magical debhelper sequence, and it almost always does the right thing for python packages
  81 [20:36] <barry> (the one exception is for python3 stuff, which we'll get to later.  you have to do some manual overrides for python3, but we're working on that)
  82 [20:36] <barry> the really important thing is the `--with-python2` option
  83 [20:37] <barry> that is what tells dh to use dh_python2 to build your package
  84 [20:37] <barry> in our case, it's really the only thing you need to add
  85 [20:37] <barry> what is `--buildsystem=python_distutils` then?
  86 [20:37] <barry> well, in this specific case, it's not required, but you will often want to add it
  87 [20:37] <barry> by default dh will ignore the setup.py if there is a Makefile there
  88 [20:38] <barry> stupid doesn't have a Makefile but many packages do, e.g. to add `make build` or `make test` targets for convenience
  89 [20:38] <barry> the --buildsystem=python_distutils tells dh to use the setup.py for various steps and ignore the Makefile
  90 [20:39] <barry> anyway, that's really all you need!  you'll notice that pkgme adds other standard debian/ files such as changelog, compat, and copyright.  that's more packaging-fu than python-packaging-fu so i'll skip over that.  i.e. none of that pertains to python packaging specifically
  91 [20:40] <barry> okay, so you should be able to take that pkgme branch, debuild -S and run pbuilder to give you a nice binary package for stupid
  92 [20:40] <barry> i'll pause for a moment for questions
  93 [20:41] <barry> okay then, moving on
  94 [20:41] <barry> remember that stupid is compatible with python2 and python3, so how would we need to modify the debian/ directory so that both versions are installed?
  95 [20:41] <barry> the first thing to understand is that in debian and ubuntu, we have completely separate stacks for python2 and python3
  96 [20:42] <barry> this means if you want a python3 version of a package, you need to install python3-foo
  97 [20:42] <barry> a good example is python-apt and python3-apt
  98 [20:42] <ClassBot> john_g asked: So what part of the work does setup.py do and what part do the dh_ things do?
  99 [20:43] <barry> setup.py does most of the work.  my recommendation is to use virtualenv and make sure your package builds, installs, and tests exactly as you want it in a python-only world (i.e. w/o debian/ubuntu getting involved)
 100 [20:43] <barry> get a solid setup.py first, using the normal python development tools.  once you have that, your debian packaging job will be *much* easier
 101 [20:44] <barry> dh_python2 does the bits to lay the package out properly within the debian file system, and to ensure that byte-compilation triggers are properly invoked when the package is installed on your system
 102 [20:44] <barry> (the .pyc files are not included in the package)
 103 [20:44] <barry> so, python3
 104 [20:45] <barry> bzr branch lp:~barry/+junk/stupid.py3
 105 [20:45] <barry> let's first look at the debian/control file
 106 [20:45] <barry> you'll noticed i fixed the Build-Depends :)
 107 [20:45] <barry> but also notice that it b-d's on both python-all-dev (for python 2) and python3-all-dev (for python3)
 108 [20:46] <barry> notice too that i've added an X-Python-Version line and an X-Python3-Version line.  this is how you control which python versions out of all that might be installed on your system, are compatible with your package
 109 [20:46] <barry> e.g. i've said that stupid is only compatible with python 3.2 and above, and python 2.6 and above
 110 [20:47] <barry> notice that i've also created two binary package stanzas, one for python-stupid and one for python3-stupid, as per the separate stack requirements
 111 [20:47] <barry> if you pull up debian/rules you'll see the additions there
 112 [20:47] <barry> we're running short on time, so i'll run through this quickly ;)
 113 [20:48] <barry> DH_VERBOSE=1 just tells dh to spew more detailed info on what it's doing.  this line is not required
 114 [20:48] <barry> the PYTHON2 and PYTHON3 lines use shell helper functions to determine which versions of python are actually installed.  we'll use these in the rules below
 115 [20:49] <barry> notice line 10, where all we've added was --with=python2,python3
 116 [20:49] <barry> that invokes dh_python3 during that part of the build process
 117 [20:49] <barry> now look at lines 13-17
 118 [20:49] <barry> i wanted to make sure that my package's tests are run during the build process, and the build should fail if the tests fail
 119 [20:50] <barry> however, i need to make sure the tests are run for every version of python we're building for
 120 [20:50] <barry> dh does not know how to do this (yet ;), so we have to add some manual rules to make this work
 121 [20:50] <barry> the test-python% lines just invoke the package's unittests with increased verbosity
 122 [20:51] <barry> the override_dh_auto_test bit is the really key for making this work, because here we're overriding dh's standard dh_auto_test call with our own.  because it depends on the test-python% target, all the tests will get invoked the way we want them to
 123 [20:51] <barry> now look at lines 20-eof
 124 [20:52] <barry> one problem we have is that dh does not yet know how to properly install the python3 built parts, so we have to do this manually
 125 [20:52] <barry> thus the override_dh_auto_install
 126 [20:52] <barry> we can just call dh_auto_install to do the right thing for python2
 127 [20:53] <barry> but then we have to manually cycle through all python3 versions and do setup.py install with some magic arguments, in order to get the python3 parts properly installed
 128 [20:53] <barry> finally, if you look in debian/ directory, you'll see two .install files.  this is how you tell the packaging toolchain which files to install for which of the multiple binary packages are getting built
 129 [20:53] <barry> look at the contents of each, and you'll see how we separate the python2 and python3 stacks
 130 [20:54] <barry> okay, i'm sorry but we've nearly run out of time.  i wish i could have covered more, but hopefully this was helpful
 131 [20:54] <barry> does anybody have any questions?
 132 [20:55] <barry> please feel free to use these three branch for cargo culting :)  i'll leave them alone and update them as the tools improve.  stupid.py3 should build fine on natty
 133 [20:55] <ClassBot> NMinker asked: how do I convert to dh_python2? Or has that been covered?
 134 [20:55] <barry> http://wiki.debian.org/Python/TransitionToDHPython2
 135 [20:56] <barry> NMinker: i've done many conversions with these instructions.  please join us on #ubuntu-pyjam for any questions after this session ends
 136 [20:56] <barry> questions or help
 137 [20:57] <barry> also, if you want to contribute to ubuntu, i can provide some packages that still need converting.  we want to remove python-central and python-support from the oneiric cds, so this is a good way to gain some packaging cred
 138 [20:58] <barry> micahg points out also this for the larger transition effort: http://people.canonical.com/~ubuntu-archive/transitions/dh-python2.html
 139 [20:59] <ClassBot> Cuzzie asked: If we want to package up the python application we wrote, we need to write all the rules, compat, control files ourselves?
 140 [20:59] <barry> Cuzzie: read the scrollback for the pkgme tool, or look into dh_make
 141 [21:00] <barry> pkgme is an excellent tool that will get you started
 142 [21:00] <barry> and with that, i think my session is done.  i will hang around and answer more questions in #ubuntu-classroom-chat