UsingAndPortingToPython3

Dev Week -- Using and porting to Python3 -- barry -- Tue, Aug 28th, 2012

   1 [19:01] <barry> hello everyone.  i'm barry warsaw and today i'll be giving a talk on python 3
   2 [19:02] <barry> this is a broad topic, and there's way too much to cover in one hour so i'll probably be going fairly quickly to cover as much as possible
   3 [19:02] <barry> it will really help if you at least already know pytho n2
   4 [19:02] <barry> *python 2
   5 [19:02] <barry> please join #ubuntu-classroom-chat and ask questions at any time!
   6 [19:03] <barry> i'll be covering some of the new features in python 3, as well as porting strategies
   7 [19:03] <barry> but i won't be talking much about porting c extensions or deb/ubu packaging (unless there's time)
   8 [19:04] <barry> i probably won't be covering much of the py3 features that are already available in py2, except to the extent that it helps with explaining how best to port stuff
   9 [19:04] <barry> i'll be providing lots of links with other resources you can read for much more detail
  10 [19:05] <ClassBot> ben72 asked: ​ is there a way to run both python 2 and 3 in the same ubuntu system?
  11 [19:06] <barry> ben72: yes.  running 'python' (aka /usr/bin/python) on any ubuntu system will run the latest version of python 2, which for ubuntu 12.04 onward is python 2.7
  12 [19:07] <barry> to run python 3 apps you need to explicitly run 'python3' or /usr/bin/python3
  13 [19:07] <barry> same for #! lines
  14 [19:07] <barry> i can talk about the difference between python 3.2, the default in ubuntu currently and python 3.3 later if you're interested
  15 [19:08] <barry> hopefully, this will get you excited about python 3 -- it's fun!
  16 [19:08] <ClassBot> NickE68 asked: ​ Will Quantal be using Python 3.3?
  17 [19:09] <barry> NickE68: no, python 3.2 will be the default python 3 version because 3.3 was not ready in time for feature freeze.  3.3 will likely be the default in 13.04
  18 [19:09] <barry> quantal does have 3.3 available though: apt-get install python3.3
  19 [19:09] <barry> and invoked via 'python3.3'
  20 [19:09] <barry> so you can play with the newest features easily
  21 [19:10] <barry> also, i fully expect that 12.10 will get a ffe for the final release of 3.3 when it comes out
  22 [19:10] <barry> so, to summarize: quantal will have 2.7, 3.2 and 3.3 with the former two as the defaults
  23 [19:10] <barry> so, why port to python 3?
  24 [19:11] <barry> there will never be a python 2.8.  i can pretty much guarantee that :)
  25 [19:11] <barry> which implies that py2 will never get any new features
  26 [19:11] <barry> and diminishing bug fixes, though the official maintenance policy is longer than usual
  27 [19:12] <barry> eventually 2.7 will only get security fixes, but i suspect upstream devs will become less interested in maintaining 2.7 as time goes on
  28 [19:12] <barry> modules will rust
  29 [19:12] <barry> py2 also has many warts, some big some small
  30 [19:12] <barry> and, most importantly imo: py3 fixes the whole bytes vs text problem
  31 [19:12] <barry> (mostly :)
  32 [19:13] <barry> meaning, we have potential to make much more robust py3 apps for non-english locales
  33 [19:13] <barry> i will talk more about bytes v. strings later
  34 [19:13] <barry> what are ubuntu's py3 plans?
  35 === Jameel is now known as Guest13536
  36 [19:14] <barry> we are on a long journey to deprecate python2.  we had hoped to port everything on the install images to py3 for 12.10, but we've hit a few big blockers so that won't happen in 12.10
  37 [19:14] <barry> maybe for 13.04, definitely for the next lts
  38 [19:15] <barry> py2.7 will never be *removed* from the archive, but it will likely only be in universe for 14.04 if we can get away with it
  39 === Guest13536 is now known as JameelA
  40 [19:16] <barry> if there are no questions related to ubuntu's py3 plans, let's talk about porting
  41 [19:16] <barry> let me give you a few links first
  42 [19:16] <barry> this is one of the very best resources available for understanding how to port your py2 code to py3:
  43 [19:16] <barry> http://python3porting.com/
  44 [19:17] <ClassBot> coalitians asked: So since the development for 12.10 is freezed, there are no ongoing porting activities now?
  45 [19:17] <barry> coalitians: we will definitely have a ffe for the gwibber backend service py3 port.  there may be a few others, but we're largely done with our porting efforts for 12.10.  from here on, it will be bug fixing
  46 [19:18] <barry> the above link talks about strategies, gives code samples for conversion of common idioms, a little bit about c extensions, and much more
  47 [19:18] <barry> i have written a number of blog posts that you might find helpful:
  48 [19:18] <barry> http://tinyurl.com/7tb3jkn
  49 [19:19] <barry> http://tinyurl.com/7famvx3
  50 [19:19] <barry> http://tinyurl.com/6ufpvfq
  51 [19:19] <barry> there's also a good page in the ubuntu wiki with specific porting help, including how to package python libraries and apps either for py3 alone, or for dual py2/py3 versions
  52 [19:20] <barry> https://wiki.ubuntu.com/Python/3
  53 [19:20] <barry> i will also give you this link to a talk from pycon 2012.  i highly recommend you watch this before you do any significant py3 work.  it is the single best description of the bytes vs. strings issue i've seen.  really a great talk:
  54 [19:20] <barry> http://pyvideo.org/video/948/pragmatic-unicode-or-how-do-i-stop-the-pain
  55 [19:21] <barry> okay, so let's talk for a moment about some porting strategies
  56 [19:22] <barry> let me say first that if you are starting a brand new python project, i would suggest going straight to py3.  one caveat is that you need to make sure any dependencies (i.e. 3rd party libraries you might want to use) are already available for py3, in ubuntu
  57 [19:22] <barry> we've worked hard these last few cycles to make sure upstreams w/ py3 support have been packaged for deb and ubu
  58 [19:22] <barry> and we've done some upstream porting work ourselves (e.g. dbus-python)
  59 [19:23] <barry> of course, not everything is available, but *a lot* is, so you're probably in great shape with quantal
  60 [19:23] <ClassBot> roadmr asked: what's the recommended path if I want to port an application using gstreamer to Python3?
  61 [19:24] <barry> roadmr: i don't have a lot of experience w/gstreamer unfortunately, so i'm not sure if a py3 version is available, and if so, where (upstream, debian, ubuntu)
  62 [19:24] <barry> i *think* it's not yet, but don't quote me on that :)
  63 [19:25] <barry> so that's for new code, what about porting existing code to py3?
  64 [19:25] <barry> ask this question: do you need to keep py2 support, and if so, how far back do you need to support?
  65 [19:25] <barry> i would *highly* recommend nothing older than py2.6 (which actually is no longer available in quantal anyway)
  66 [19:25] <barry> and would recommend at least py2.7 for some very useful features (e.g. the new unittest library)
  67 [19:26] <barry> the thing is: py2.7 and to some extent 2.6 has lots of backported features that can make porting, and supporting py2 and py3 much easier
  68 [19:26] <barry> through things like "from __future__ import"s
  69 [19:27] <barry> so, when starting a porting effort, the first thing to do is to run your code with 'python2.7 -3'
  70 [19:27] <barry> that -3 flag will cause python to warn you about any idiom that cannot be easily ported with the 2to3 program (more on that later)
  71 [19:27] <barry> if your py2 program runs cleanly with 'python2.7 -3' then you're in pretty good shape for starting your py3 port
  72 [19:28] <barry> 2to3 is a code transformation framework that can take most common py2 idioms and convert them on the fly to py3
  73 [19:28] <barry> my personal opinion is that its output is useful but shouldn't be used "in production".  it's a bit too slow in the dev cycle for me.
  74 [19:28] <barry> but it's pretty flexible
  75 [19:29] <barry> my own recommendation is to use a single code base and just be careful about how you code.  use from-future imports as much as possible
  76 [19:29] <barry> also there's a third party 'six' library (apt-get install python-six i believe) that can make a single code base easier to maintain
  77 [19:30] <barry> but here's the most important thing you need to do before you port, and it segues into my next topic:
  78 [19:30] <barry> you must be *crystal clear* about the distinction in your data model between bytes (i.e. binary data) and strings (i.e. human readable text)
  79 [19:31] <barry> if you are getting lots of UnicodeErrors in your py2 code, it's a sign that you are not crystal clear :)
  80 [19:31] <barry> and that is one of the biggest benefits (and frankly headaches while porting) of py3.  the requirement to be clear means that once you've done so, your non-english users won't be getting UnicodeErrors
  81 [19:32] <ClassBot> BlessJah asked: is it possible to write to write py3 code that would run also in py2 environment?
  82 [19:32] <barry> BlessJah: that's not really much different than writing py2 code that runs in a py3 environment.  yes, its very definitely possible
  83 [19:33] <barry> so, hopefully everyone understand the difference between bytes and strings
  84 [19:33] <barry> bytes are binary data, e.g. possibly images, audio, stuff like that
  85 [19:33] <barry> strings are human readable text
  86 [19:33] <barry> in py2 you had two "string" types, str which were also called 8-bit strings and they served as ascii text and binary data
  87 [19:34] <barry> and unicodes which are just that, unicode human readable text
  88 [19:34] <barry> the big py2 problem was that python auto-coerced between the two, often in ways that were not correct, leading to dreaded UnicodeErrors
  89 [19:34] <barry> in py3 you have bytes which are *only* binary data
  90 [19:34] <barry> and str which are unicodes
  91 [19:35] <barry> and there is no auto-conversion
  92 [19:35] <barry> you cannot concatenate the two without explicitly converting
  93 [19:35] <barry> so if you have a unicode string, you can encode it to bytes, but you must know the encoding (e.g. utf-8 usually, but maybe latin-1 or any of the other multitude of encodings available)
  94 [19:36] <barry> to go from bytes to str, you must decode, again by being explicit about the encoding
  95 [19:36] <barry> an important thing to remember now too is that in py3, by default open() will open a file in binary mode and you read bytes out of it, *not* text
  96 [19:37] <barry> unlike in py2, py3's open() takes an encoding argument, so if you call it like so: open('myfile', encoding='utf-8')
  97 [19:37] <barry> you will be reading unicode strings from the file
  98 [19:37] <barry> which is very nice (assuming of course that the contents of your file is utf-8 encoded strings :)
  99 [19:38] <barry> in py3, there is no more u'' prefix for strings.  unadorned string literals are unicodes
 100 === keffie_jayx is now known as effiejayx
 101 [19:38] <barry> (aside: the u'' prefix is reintroduced in py3.3 but essentially noops)
 102 [19:38] <barry> to write bytes literals, you use the b'' prefix
 103 [19:38] <barry> e.g. this is bytes: b'foo'
 104 [19:38] <barry> and this is str: 'foo'
 105 [19:39] <barry> one other quick note about this: you can now safely use the gettext module in py3 since it always returns unicodes from catalog lookups :)
 106 [19:39] <barry> are there any questions about bytes vs strings before i move on?
 107 [19:40] <ClassBot> NickE68 asked: ​ What if you don't know the encoding used for some file? Is there lib that can produce a 'best guess'?
 108 [19:41] <barry> NickE68: there is, but i'm blanking on the name of it.  i believe it comes out of mozilla.  in general, it's much safer to know
 109 [19:41] <barry> you can also try some of the most common ones, e.g. utf-8 first, then latin-1
 110 [19:41] <barry> then ascii
 111 [19:41] <ClassBot> roadmr asked: gettext, "it always returns unicodes". Do you mean "strings" here, as opposed to "bytes"?
 112 [19:42] <barry> roadmr: yes.  in py2 unfortunately, gettext.gettext('foo') always returned an 8-bit string, which really causes problems, and the api for getting (py2) unicodes is pretty well hidden and not the default
 113 [19:42] <barry> in py3, gettext.gettext('foo') always returns a str, aka a unicode
 114 [19:43] <barry> jtaylor: reminds me that it's python-chardet.  thanks!
 115 [19:43] <barry> cool. so, in the remaining 17m or so, i'm going to give a whirlwind tour of some of the changes and new features in py3 to be aware of
 116 [19:43] <barry> this is by no means exhaustive!
 117 [19:44] <barry> there are "what's new" documents out there for each individual version of python 3, but afaik, no overarching list of differences b/w say py2.7 and py3.2
 118 [19:44] <barry> though the pythonporting url i gave above comes pretty close
 119 [19:45] <barry> so, let's begin, and please ask about further details at any point
 120 [19:45] <barry> many parts of the stdlib have been reorganized
 121 [19:45] <barry> this means that some old, unsupported modules have been removed
 122 [19:46] <barry> probably nothing you care about unless you're doing image processing on irix and then why are you here? :)
 123 [19:46] <barry> many modules have been moved around, some placed inside of packages
 124 [19:46] <barry> e.g. the very common 'urlparse()' function now lives in urllib.parse.urlparse()
 125 [19:47] <barry> http://docs.python.org/py3k/library/ is your best friend
 126 [19:47] <barry> some builtins have been removed or moved
 127 [19:47] <barry> e.g. the reload() builtin has been moved to imp.reload()
 128 [19:48] <barry> for the builtins that have been removed, there's usually a good alternative
 129 [19:48] <barry> e.g. callable() is gone, but you can test for callability either by just calling it <wink> or through the inspect module
 130 [19:49] <barry> one big change that gave me much finger pain until i re-taught the muscle memory (and started using "from __future__ import print_function" in my py2 code)
 131 [19:49] <barry> is that the print statement has become a function
 132 [19:49] <barry> e.g. print('hello world')
 133 [19:49] <barry> this was very jarring, but it makes great sense, and eventually you'll come to appreciate this change :)
 134 [19:50] <barry> you know how python2 has essentially two types of classes?  classic classes and "new style" classes?
 135 [19:50] <ClassBot> There are 10 minutes remaining in the current session.
 136 [19:50] <barry> py3 gets rid of classic classes.  all classes in py3 are "new style"
 137 [19:50] <barry> so that means you no longer have to inherit from object
 138 [19:51] <barry> or use the "__metaclass__ = type" trick to get all the great features of "new style" classes
 139 [19:51] <barry> i guess that means we have to stop calling them "new style" :)
 140 [19:51] <barry> py2 has two types of integer, int and long
 141 [19:51] <barry> py3 does away with this and has only one integer type, int
 142 [19:52] <barry> it automatically knows to use the more efficient internal representation for smaller ints and the expensive representation for bigger ints
 143 [19:52] <barry> there is no more 'L' suffix on int literals
 144 [19:52] <ClassBot> jsjgruber-l82-p asked: Any high precision float types in py3?
 145 [19:53] <barry> jsjgruber-x-p_: i think so, but i do so little math in python, i can't help much with that
 146 [19:53] <barry> i know there's a decimal type (though it's not built-in)
 147 [19:54] <barry> the dict.iter*() methods are gone in py3. i.e. no more .iterkeys(), .itervalues(), or .iteritems()
 148 [19:54] <barry> just use .keys() et al, and remember that they return iterators in py3 *not* concrete lists
 149 [19:54] <barry> you might need to wrap the return value in list(dict.values()) for example
 150 [19:55] <barry> the syntax for catching exceptions has changed (though also available in py2.6 and 2.7)
 151 [19:55] <barry> gone is "except Foo, e"
 152 [19:55] <barry> use
 153 [19:55] <barry> "except Foo as e" instead
 154 [19:55] <barry> instead of the exec statement, use the exec() function
 155 [19:55] <ClassBot> There are 5 minutes remaining in the current session.
 156 [19:55] <barry> file() builtin is gone, use open()
 157 [19:56] <barry> no more xrange().  go back to range() which now returns an iterator
 158 [19:56] <barry> as does filter() and map()
 159 [19:56] <barry> (again, you might need to wrap them in list() to get a concrete list)
 160 [19:56] <barry> absolute imports are the default, you must be explicit about relative imports
 161 [19:57] <barry> okay, there's no way i'm going to finish. so are there any last minute questions?
 162 [19:57] <barry> :)
 163 [19:57] <barry> i'll just keep typing until classbot shuts me up
 164 [19:57] <barry> 1 / 2 returns 0.5 now
 165 [19:57] <barry> use 1//2 to get the old floor division behavior
 166 [19:57] <ClassBot> coalitians asked: Barry, How and where should we get started to be part of ubuntu python community?
 167 [19:58] <barry> coalitians: i would highly suggest joining the debian python community over on OFTC #debian-python.  #ubuntu-devel is good too, as is #python on freenode
 168 [19:58] <barry> there isn't a dedicated ubuntu python forum though.  maybe there should be!
 169 [19:59] <ClassBot> NickE68 asked: ​ Is there a place listing packages for which porting help is required (for 13.04 now, I guess)?
 170 [19:59] <barry> NickE68: yes:
 171 [19:59] <barry> https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AiT4gOXSkmapdFA1anRkWERsaXgtWnllUG9QWXhDVWc&pli=1#gid=0
 172 [19:59] <barry> https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-python-versions
 173 [19:59] <barry> https://wiki.ubuntu.com/Python/FoundationsQPythonVersions
 174 [20:00] <barry> please ping me in #ubuntu-devel for more info.  we'd *love* the help!

MeetingLogs/devweek1208/UsingAndPortingToPython3 (last edited 2012-08-29 10:46:01 by dholbach)