Ubuntu Wiki

   1 [20:05] <DktrKranz> Hello everybody, and thank you to be here at Python Application packaging hosted at Ubuntu Developer Week!
   2 [20:05] <DktrKranz> My name is Luca Falavigna, I'm a MOTU and Debian Developer, and I'll be your SABPDFT (Self-Appointed Benevolent Python Dictator For Today) for the next minutes :)
   3 [20:06] <DktrKranz> For those who weren't here last week, I already had a session about basic Python application packaging. You can find logs here: http://irclogs.ubuntu.com/2010/01/21/%23ubuntu-classroom.html
   4 [20:06] <DktrKranz> It basically covered how to handle control files in debian/, and how to easily find dependencies for your Python applications. Even if you are good in Python packaging, there could be some interesting tips to improve your kung-fu :)
   5 [20:07] <DktrKranz> Today we will look at some advanced tricks to improve overall Python packages quality and stability.
   6 [20:07] <DktrKranz> But let's start with a really short introduction.
   7 [20:08] <DktrKranz> I refer to a "Python application" as a piece of software composed by one or more Python scripts and eventually one of more modules (or "packages", distutils calls them that way).
   8 [20:08] <DktrKranz> Other kinds of software are "Python modules", which contain modules potentially useful for other programs installed in the global namespace, and "Python extensions", which usually are C source files, compiled and liked for a given Python interpreter to extend its features.
   9 [20:09] <DktrKranz> But how does a Python application look like?
  10 [20:09] <DktrKranz> I've created a really dumb one for the occasion. Open your favourite terminal and launch dget -u http://people.debian.org/~dktrkranz/pythontest/pythontest_0.1-1/pythontest_0.1-1.dsc
  11 [20:10] <DktrKranz> Once you've downloaded it, look at pythontest-0.1 directory, you will find pythontest script, PythonTest module (or "package") and setup.py, which is Python distutils' "makefile".
  12 [20:11] <DktrKranz> It's probably the smallest Python application you'll ever seen, it just prints a message, and then exits :)
  13 [20:11] <DktrKranz> Invest a couple of minutes to examine the few files available, then we will move to packaging.
  14 [20:12] <DktrKranz> < Omar871> QUESTION: Will "setup.py install" work?
  15 [20:12] <DktrKranz> Omar871: it should by launching "python setup.py install"
  16 [20:13] <DktrKranz> I didn't test that way, but it should work as expected ;)
  17 [20:15] <DktrKranz> < Omar871> QUESTION: How do I use the package after installing it?
  18 [20:16] <DktrKranz> it installs a script under /usr/bin, named pythontest. you should be able to launch it, but we will see it in the next minutes
  19 [20:16] <DktrKranz> < statik-lernid> QUESTION: when would we choose dh --with-python-central instead of the default?
  20 [20:17] <DktrKranz> statik-lernid: it's a matter of tastes. I personally prefer python-support, I read the code and I understand a bit its logic.
  21 [20:18] <DktrKranz> < lucio> QUESTION: what is pycentrak and pysupport and why do i care?
  22 [20:19] <DktrKranz> lucio: these are helpers which manage to expand ${python:Depends} to the right values, and byte-compile Python files during package install
  23 [20:20] <DktrKranz> they do more, but that's beyond the scope of the lesson.
  24 [20:20] <DktrKranz> lucio: if you are interested, a more detailed description of packaging is held in the logs of the other session, see above :)
  25 [20:21] <DktrKranz> ok, let's move to the packaging itself now
  26 [20:21] <DktrKranz> Compile the package (dpkg-buildpackage -uc -us) and install the resulting pythontest_0.1-1_all.deb
  27 [20:22] <DktrKranz> To test it, open another console and move to a completely different location than the build directory. Keep this in mind: never test a Python software installed system-wide by launching it in the build directory. This will save you from errors, and let you discover serious problems in your packaging!
  28 [20:23] <DktrKranz> Now launch "pythontest". You should now see a message on your console saying package works correctly.
  29 [20:25] <DktrKranz> < Omar871> QUESTION: Is debhelper needed for this?
  30 [20:25] <DktrKranz> Omar871: for this examples, yes. debhelper is explicitly listed in Depends field in debian/control
  31 [20:27] <DktrKranz> < statik-lernid> QUESTION: when I dpkg -L pythontest, I see files installed to /usr/share/pyshared/. when I python -c "import PythonTest;print
  32 [20:27] <DktrKranz>                        PythonTest.__file__" I see files located at /usr/lib/pymodules/python2.6/PythonTest/__init__.pyc. why the different paths?
  33 [20:28] <DktrKranz> statik-lernid: good question. This is how python-support worsk
  34 [20:28] <DktrKranz> it byte-compiles files for every supported Python version at runtime, and stores files under /usr/lib/pymodules/python*/PythonTest
  35 [20:29] <DktrKranz> while package only has them under pyshared, because they are common to multiple Python versions at the same time
  36 [20:29] <DktrKranz> that's the magic behind python-support and python-central :)
  37 [20:30] <DktrKranz> < POX> QUESTION: shouldn't Python applications use private dirs by default (i.e. when other packages do not use their modules)? What's --install-lib
  38 [20:30] <DktrKranz> and --install-scripts?
  39 [20:30] <DktrKranz> POX: yes. Stay tuned for that :)
  40 [20:31] <DktrKranz> Even if package works, we installed PythonTest module in Python global namespace, which is completely useless for other packages in this case. Nobody will ever want to access our PythonTest module, and we should also want to avoid collisions with other modules as well.
  41 [20:31] <DktrKranz> To avoid global namespace pollution, we should install our PythonTest module in a "private directory", that is one not listed in the output shown by python -c "import sys; print sys.path" command.
  42 [20:32] <DktrKranz> /usr/share/packagename is a very good choice in almost all cases, so we'll adjust our package to install modules in /usr/share/pythontest.
  43 [20:33] <DktrKranz> I prepared a second package, you can grab it with dget -u http://people.debian.org/~dktrkranz/pythontest/pythontest_0.1-2/pythontest_0.1-2.dsc, and see changes I've made here: http://people.debian.org/~dktrkranz/pythontest/0.1-1_0.1-2.patch
  44 [20:34] <DktrKranz> Basically, the only change I did is passing --install-lib=/usr/share/pythontest to distutils to instruct it to install PythonTest in our private directory (see the three rules files to see several methods to do that).
  45 [20:34] <DktrKranz> Again, compile, install the resulting pythontest_0.1-2_all.deb, and test it by launching pythontest in the other console.
  46 [20:34] <DktrKranz> You should now see something unexpected...
  47 [20:35] <DktrKranz> I got "Uh-oh! There is something wrong with the package, I can not import PythonTest module!", so we probably did something bad here :)
  48 [20:37] <DktrKranz> Before explaining that, let's answer to some questions.
  49 [20:37] <DktrKranz> < mhall119|work> QUESTION: so the dh $@ in debian/rules is processing the pythontest.* files?
  50 [20:38] <DktrKranz> mhall119|work: it's dh_pysupport (or dh_pycentral), called by dh sequencer which parsed "dh $@"
  51 [20:39] <DktrKranz> you can see dh_pysupport has been called in your terminal output
  52 [20:40] <DktrKranz> < Omar871> QUESTION: What about virtual env?
  53 [20:40] <DktrKranz> Omar871: we don't usually use virtualenv to avoid shipping multiple files in the same package
  54 [20:41] <DktrKranz> < mhall119|work> QUESTION: so I understand, dh->dh_pysupport->pythontest.postinst.debhelper
  55 [20:42] <DktrKranz> mhall119|work: exactly. dh calls dh_pysupport at the given time, while dh_pysupport also generates *postinst files to handle byte-compilation of Python objects.
  56 [20:43] <DktrKranz> < johnsgruber> QUESTION: In my case for the first example source package the built deb says to put the library in /usr/share/python-support/pythontest
  57 [20:43] <DktrKranz> johnsgruber: do you have a python-support version << 0.90.0 ?
  58 [20:44] <DktrKranz> e.g, you use an older Ubuntu/Debian release than Intrepid/Karmic, or testing/sid?
  59 [20:46] <DktrKranz> Back to our problem then.
  60 [20:46] <DktrKranz> Why did we get that? We moved PythonTest to a private directory, and this is good, but our pythontest script is no longer able to access it, and this is bad!
  61 [20:46] <DktrKranz> This is why PythonTest is not in global namespace anymore, but pythontest script obviously thinks it is there.
  62 [20:47] <DktrKranz> We have two options now: using sys.path.append() from inside of pythontest script (the one installed in /usr/bin), or moving it to the same directory where PythonTest is (/usr/share/pythontest).
  63 [20:48] <DktrKranz> My preferred way to achieve this goal is the latter, I don't like playing with sys.path :)
  64 [20:48] <DktrKranz> Let's download today's third package with dget -u http://people.debian.org/~dktrkranz/pythontest/pythontest_0.1-3/pythontest_0.1-3.dsc and look at the diff at http://people.debian.org/~dktrkranz/pythontest/0.1-2_0.1-3.patch
  65 [20:49] <DktrKranz> This time I passed a new option to distutils: --install-script=/usr/share/pythontest, to instruct it to install our scripts in /usr/share/pythontest too. I also created a /usr/bin/pythontest symlink pointing to /usr/share/pythontest/pythontest, this way we still have pythontest in our $PATH :)
  66 [20:49] <DktrKranz> Again, compile, install the resulting pythontest_0.1-3_all.deb and test it by launching pythontest in the other console.
  67 [20:50] <DktrKranz> Now you should see something familiar, and more friendly :)
  68 [20:51] <DktrKranz> We have PythonTest module in a private directory, and pythontest is able to import it without problem, let's celebrate! \o/
  69 [20:51] <DktrKranz> It's a little tricky at times, but you just need some practice, and things will be clearer.
  70 [20:51] <DktrKranz> < mhall119|work> QUESTION: applications should always be put in a private space?
  71 [20:52] <DktrKranz> mhall119|work: it's not always possible, sometimes modules are required by other packages, and having it in global namespace helps, but normally the private directory should be used
  72 [20:53] <DktrKranz> in my memory, > 95% Python applications can be installed using private directory method
  73 [20:54] <DktrKranz> < Omar871> QUESTION: Can you please explain to me what exactly is a "private space"?
  74 [20:55] <DktrKranz> Omar871: a private directory is a directory not accessible by Python global namespace.
  75 [20:56] <DktrKranz> Omar871: for example, if you launch python console and import a module (f.e. gtk, but feel free to pick another one), python interpreter will look in some directories looing for gtk module
  76 [20:56] <DktrKranz> these are the ones listed by "python -c "import sys; print sys.path" command
  77 [20:57] <DktrKranz> other ones are "private", given that python interpreter won't look in those searching for modules
  78 [20:57] <DktrKranz> < POX> QUESTION: what if script and module name is the same? What's --install-scripts?
  79 [20:58] <DktrKranz> POX: in that case, one of the quickest solutions is to manually install scripts having the same name as modules/packages (e.g. foo -> foo.py)
  80 [20:59] <DktrKranz> POX: --install-scripts is another distutils option which instructs it to install files listed under scripts=[] (seen it in setup.py) to a given directory.
  81 [21:00] <DktrKranz> < rmunn|lernid> QUESTION: Won't DktrKranz's approach fail once Python 2.7 is released and absolute imports (http://www.python.org/dev/peps/pep-0328/)
  82 [21:00] <DktrKranz> become the default? Isn't it better to fiddle with sys.path?
  83 [21:02] <DktrKranz> rmunn|lernid: nice question. I haven't had the occasion to look at relative and absolute imports yet applied to Debian packaging, but I guess something have to be adjusted for that, and sys.path mangling could become handy.
  84 [21:03] <DktrKranz> < mhall119|work> QUESTION: is rules.cdbs being used in what we're currently doing?  I see that your patches updated it as well
  85 [21:03] <DktrKranz> mhall119|work: not directly. I included it in my example for those familiar with CDBS over dh7. I patched it to show how to handle things with different helpers.
  86 [21:04] <DktrKranz> (debian/rules.olddh follows the same rule)
  87 [21:06] <DktrKranz> < mhall119|work> QUESTION: is setup.py being called by anything in this setup?
  88 [21:06] <DktrKranz> mhall119|work: it's called by dh_auto_install
  89 [21:07] <DktrKranz> which in turn is called by dh sequencer, yay for tiny rules :)
  90 [21:08] <DktrKranz> < Omar871> QUESTION: Which channel can we find you at?
  91 [21:09] <DktrKranz> Omar871: several maintainer and developers interested in Python packaging are on #debian-python, OFTC network
  92 [21:09] <DktrKranz> feel free to join, ask questions and look for sponsorship :)
  93 [21:12] <DktrKranz> < Emilien> QUESTION: elaboration: Well, I've heard that with Quickly you can package your app, which is written in Python too... Wouldn't it be easier
  94 [21:12] <DktrKranz>                  to use that instead of having to set everything manually up?
  95 [21:13] <DktrKranz> Emilien: I've heard of Quicly, I know is active developed by some cool guys here, I didn't have the occasion to test it yet, I still prefer using my favourite editor, just because things to cut are huge.
  96 [21:14] <DktrKranz> so, I can't answer to that question, but I guess if it manages things as described here today, it should be fine (but please don't completely trust helpers)
  97 [21:15] <DktrKranz> < Emilien> dktrkranz: what do you mean by "things to cut are huge"?
  98 [21:16] <DktrKranz> Emilien: usually, templates are prepared to fit everyone's needs, while I prefer to tailor a package to fit its needs, removing every unneeded bit.
  99 [21:19] <DktrKranz> OK, we're a bit late, if you still have questions, we can move to our usual channels. Thanks everyone for coming, I had a great time! See you :)