Dev Week -- Using and porting to Python3 -- barry -- Tue, Aug 28th, 2012
1 [19:01] <barry> hello everyone. i'm barry warsaw and today i'll be giving a talk on python 3 2 [19:02] <barry> this is a broad topic, and there's way too much to cover in one hour so i'll probably be going fairly quickly to cover as much as possible 3 [19:02] <barry> it will really help if you at least already know pytho n2 4 [19:02] <barry> *python 2 5 [19:02] <barry> please join #ubuntu-classroom-chat and ask questions at any time! 6 [19:03] <barry> i'll be covering some of the new features in python 3, as well as porting strategies 7 [19:03] <barry> but i won't be talking much about porting c extensions or deb/ubu packaging (unless there's time) 8 [19:04] <barry> i probably won't be covering much of the py3 features that are already available in py2, except to the extent that it helps with explaining how best to port stuff 9 [19:04] <barry> i'll be providing lots of links with other resources you can read for much more detail 10 [19:05] <ClassBot> ben72 asked: â€‹â€†is there a way to run both python 2 and 3 in the same ubuntu system? 11 [19:06] <barry> ben72: yes. running 'python' (aka /usr/bin/python) on any ubuntu system will run the latest version of python 2, which for ubuntu 12.04 onward is python 2.7 12 [19:07] <barry> to run python 3 apps you need to explicitly run 'python3' or /usr/bin/python3 13 [19:07] <barry> same for #! lines 14 [19:07] <barry> i can talk about the difference between python 3.2, the default in ubuntu currently and python 3.3 later if you're interested 15 [19:08] <barry> hopefully, this will get you excited about python 3 -- it's fun! 16 [19:08] <ClassBot> NickE68 asked: â€‹â€†Will Quantal be using Python 3.3? 17 [19:09] <barry> NickE68: no, python 3.2 will be the default python 3 version because 3.3 was not ready in time for feature freeze. 3.3 will likely be the default in 13.04 18 [19:09] <barry> quantal does have 3.3 available though: apt-get install python3.3 19 [19:09] <barry> and invoked via 'python3.3' 20 [19:09] <barry> so you can play with the newest features easily 21 [19:10] <barry> also, i fully expect that 12.10 will get a ffe for the final release of 3.3 when it comes out 22 [19:10] <barry> so, to summarize: quantal will have 2.7, 3.2 and 3.3 with the former two as the defaults 23 [19:10] <barry> so, why port to python 3? 24 [19:11] <barry> there will never be a python 2.8. i can pretty much guarantee that :) 25 [19:11] <barry> which implies that py2 will never get any new features 26 [19:11] <barry> and diminishing bug fixes, though the official maintenance policy is longer than usual 27 [19:12] <barry> eventually 2.7 will only get security fixes, but i suspect upstream devs will become less interested in maintaining 2.7 as time goes on 28 [19:12] <barry> modules will rust 29 [19:12] <barry> py2 also has many warts, some big some small 30 [19:12] <barry> and, most importantly imo: py3 fixes the whole bytes vs text problem 31 [19:12] <barry> (mostly :) 32 [19:13] <barry> meaning, we have potential to make much more robust py3 apps for non-english locales 33 [19:13] <barry> i will talk more about bytes v. strings later 34 [19:13] <barry> what are ubuntu's py3 plans? 35 === Jameel is now known as Guest13536 36 [19:14] <barry> we are on a long journey to deprecate python2. we had hoped to port everything on the install images to py3 for 12.10, but we've hit a few big blockers so that won't happen in 12.10 37 [19:14] <barry> maybe for 13.04, definitely for the next lts 38 [19:15] <barry> py2.7 will never be *removed* from the archive, but it will likely only be in universe for 14.04 if we can get away with it 39 === Guest13536 is now known as JameelA 40 [19:16] <barry> if there are no questions related to ubuntu's py3 plans, let's talk about porting 41 [19:16] <barry> let me give you a few links first 42 [19:16] <barry> this is one of the very best resources available for understanding how to port your py2 code to py3: 43 [19:16] <barry> http://python3porting.com/ 44 [19:17] <ClassBot> coalitians asked: So since the development for 12.10 is freezed, there are no ongoing porting activities now? 45 [19:17] <barry> coalitians: we will definitely have a ffe for the gwibber backend service py3 port. there may be a few others, but we're largely done with our porting efforts for 12.10. from here on, it will be bug fixing 46 [19:18] <barry> the above link talks about strategies, gives code samples for conversion of common idioms, a little bit about c extensions, and much more 47 [19:18] <barry> i have written a number of blog posts that you might find helpful: 48 [19:18] <barry> http://tinyurl.com/7tb3jkn 49 [19:19] <barry> http://tinyurl.com/7famvx3 50 [19:19] <barry> http://tinyurl.com/6ufpvfq 51 [19:19] <barry> there's also a good page in the ubuntu wiki with specific porting help, including how to package python libraries and apps either for py3 alone, or for dual py2/py3 versions 52 [19:20] <barry> https://wiki.ubuntu.com/Python/3 53 [19:20] <barry> i will also give you this link to a talk from pycon 2012. i highly recommend you watch this before you do any significant py3 work. it is the single best description of the bytes vs. strings issue i've seen. really a great talk: 54 [19:20] <barry> http://pyvideo.org/video/948/pragmatic-unicode-or-how-do-i-stop-the-pain 55 [19:21] <barry> okay, so let's talk for a moment about some porting strategies 56 [19:22] <barry> let me say first that if you are starting a brand new python project, i would suggest going straight to py3. one caveat is that you need to make sure any dependencies (i.e. 3rd party libraries you might want to use) are already available for py3, in ubuntu 57 [19:22] <barry> we've worked hard these last few cycles to make sure upstreams w/ py3 support have been packaged for deb and ubu 58 [19:22] <barry> and we've done some upstream porting work ourselves (e.g. dbus-python) 59 [19:23] <barry> of course, not everything is available, but *a lot* is, so you're probably in great shape with quantal 60 [19:23] <ClassBot> roadmr asked: what's the recommended path if I want to port an application using gstreamer to Python3? 61 [19:24] <barry> roadmr: i don't have a lot of experience w/gstreamer unfortunately, so i'm not sure if a py3 version is available, and if so, where (upstream, debian, ubuntu) 62 [19:24] <barry> i *think* it's not yet, but don't quote me on that :) 63 [19:25] <barry> so that's for new code, what about porting existing code to py3? 64 [19:25] <barry> ask this question: do you need to keep py2 support, and if so, how far back do you need to support? 65 [19:25] <barry> i would *highly* recommend nothing older than py2.6 (which actually is no longer available in quantal anyway) 66 [19:25] <barry> and would recommend at least py2.7 for some very useful features (e.g. the new unittest library) 67 [19:26] <barry> the thing is: py2.7 and to some extent 2.6 has lots of backported features that can make porting, and supporting py2 and py3 much easier 68 [19:26] <barry> through things like "from __future__ import"s 69 [19:27] <barry> so, when starting a porting effort, the first thing to do is to run your code with 'python2.7 -3' 70 [19:27] <barry> that -3 flag will cause python to warn you about any idiom that cannot be easily ported with the 2to3 program (more on that later) 71 [19:27] <barry> if your py2 program runs cleanly with 'python2.7 -3' then you're in pretty good shape for starting your py3 port 72 [19:28] <barry> 2to3 is a code transformation framework that can take most common py2 idioms and convert them on the fly to py3 73 [19:28] <barry> my personal opinion is that its output is useful but shouldn't be used "in production". it's a bit too slow in the dev cycle for me. 74 [19:28] <barry> but it's pretty flexible 75 [19:29] <barry> my own recommendation is to use a single code base and just be careful about how you code. use from-future imports as much as possible 76 [19:29] <barry> also there's a third party 'six' library (apt-get install python-six i believe) that can make a single code base easier to maintain 77 [19:30] <barry> but here's the most important thing you need to do before you port, and it segues into my next topic: 78 [19:30] <barry> you must be *crystal clear* about the distinction in your data model between bytes (i.e. binary data) and strings (i.e. human readable text) 79 [19:31] <barry> if you are getting lots of UnicodeErrors in your py2 code, it's a sign that you are not crystal clear :) 80 [19:31] <barry> and that is one of the biggest benefits (and frankly headaches while porting) of py3. the requirement to be clear means that once you've done so, your non-english users won't be getting UnicodeErrors 81 [19:32] <ClassBot> BlessJah asked: is it possible to write to write py3 code that would run also in py2 environment? 82 [19:32] <barry> BlessJah: that's not really much different than writing py2 code that runs in a py3 environment. yes, its very definitely possible 83 [19:33] <barry> so, hopefully everyone understand the difference between bytes and strings 84 [19:33] <barry> bytes are binary data, e.g. possibly images, audio, stuff like that 85 [19:33] <barry> strings are human readable text 86 [19:33] <barry> in py2 you had two "string" types, str which were also called 8-bit strings and they served as ascii text and binary data 87 [19:34] <barry> and unicodes which are just that, unicode human readable text 88 [19:34] <barry> the big py2 problem was that python auto-coerced between the two, often in ways that were not correct, leading to dreaded UnicodeErrors 89 [19:34] <barry> in py3 you have bytes which are *only* binary data 90 [19:34] <barry> and str which are unicodes 91 [19:35] <barry> and there is no auto-conversion 92 [19:35] <barry> you cannot concatenate the two without explicitly converting 93 [19:35] <barry> so if you have a unicode string, you can encode it to bytes, but you must know the encoding (e.g. utf-8 usually, but maybe latin-1 or any of the other multitude of encodings available) 94 [19:36] <barry> to go from bytes to str, you must decode, again by being explicit about the encoding 95 [19:36] <barry> an important thing to remember now too is that in py3, by default open() will open a file in binary mode and you read bytes out of it, *not* text 96 [19:37] <barry> unlike in py2, py3's open() takes an encoding argument, so if you call it like so: open('myfile', encoding='utf-8') 97 [19:37] <barry> you will be reading unicode strings from the file 98 [19:37] <barry> which is very nice (assuming of course that the contents of your file is utf-8 encoded strings :) 99 [19:38] <barry> in py3, there is no more u'' prefix for strings. unadorned string literals are unicodes 100 === keffie_jayx is now known as effiejayx 101 [19:38] <barry> (aside: the u'' prefix is reintroduced in py3.3 but essentially noops) 102 [19:38] <barry> to write bytes literals, you use the b'' prefix 103 [19:38] <barry> e.g. this is bytes: b'foo' 104 [19:38] <barry> and this is str: 'foo' 105 [19:39] <barry> one other quick note about this: you can now safely use the gettext module in py3 since it always returns unicodes from catalog lookups :) 106 [19:39] <barry> are there any questions about bytes vs strings before i move on? 107 [19:40] <ClassBot> NickE68 asked: â€‹â€†What if you don't know the encoding used for some file? Is there lib that can produce a 'best guess'? 108 [19:41] <barry> NickE68: there is, but i'm blanking on the name of it. i believe it comes out of mozilla. in general, it's much safer to know 109 [19:41] <barry> you can also try some of the most common ones, e.g. utf-8 first, then latin-1 110 [19:41] <barry> then ascii 111 [19:41] <ClassBot> roadmr asked: gettext, "it always returns unicodes". Do you mean "strings" here, as opposed to "bytes"? 112 [19:42] <barry> roadmr: yes. in py2 unfortunately, gettext.gettext('foo') always returned an 8-bit string, which really causes problems, and the api for getting (py2) unicodes is pretty well hidden and not the default 113 [19:42] <barry> in py3, gettext.gettext('foo') always returns a str, aka a unicode 114 [19:43] <barry> jtaylor: reminds me that it's python-chardet. thanks! 115 [19:43] <barry> cool. so, in the remaining 17m or so, i'm going to give a whirlwind tour of some of the changes and new features in py3 to be aware of 116 [19:43] <barry> this is by no means exhaustive! 117 [19:44] <barry> there are "what's new" documents out there for each individual version of python 3, but afaik, no overarching list of differences b/w say py2.7 and py3.2 118 [19:44] <barry> though the pythonporting url i gave above comes pretty close 119 [19:45] <barry> so, let's begin, and please ask about further details at any point 120 [19:45] <barry> many parts of the stdlib have been reorganized 121 [19:45] <barry> this means that some old, unsupported modules have been removed 122 [19:46] <barry> probably nothing you care about unless you're doing image processing on irix and then why are you here? :) 123 [19:46] <barry> many modules have been moved around, some placed inside of packages 124 [19:46] <barry> e.g. the very common 'urlparse()' function now lives in urllib.parse.urlparse() 125 [19:47] <barry> http://docs.python.org/py3k/library/ is your best friend 126 [19:47] <barry> some builtins have been removed or moved 127 [19:47] <barry> e.g. the reload() builtin has been moved to imp.reload() 128 [19:48] <barry> for the builtins that have been removed, there's usually a good alternative 129 [19:48] <barry> e.g. callable() is gone, but you can test for callability either by just calling it <wink> or through the inspect module 130 [19:49] <barry> one big change that gave me much finger pain until i re-taught the muscle memory (and started using "from __future__ import print_function" in my py2 code) 131 [19:49] <barry> is that the print statement has become a function 132 [19:49] <barry> e.g. print('hello world') 133 [19:49] <barry> this was very jarring, but it makes great sense, and eventually you'll come to appreciate this change :) 134 [19:50] <barry> you know how python2 has essentially two types of classes? classic classes and "new style" classes? 135 [19:50] <ClassBot> There are 10 minutes remaining in the current session. 136 [19:50] <barry> py3 gets rid of classic classes. all classes in py3 are "new style" 137 [19:50] <barry> so that means you no longer have to inherit from object 138 [19:51] <barry> or use the "__metaclass__ = type" trick to get all the great features of "new style" classes 139 [19:51] <barry> i guess that means we have to stop calling them "new style" :) 140 [19:51] <barry> py2 has two types of integer, int and long 141 [19:51] <barry> py3 does away with this and has only one integer type, int 142 [19:52] <barry> it automatically knows to use the more efficient internal representation for smaller ints and the expensive representation for bigger ints 143 [19:52] <barry> there is no more 'L' suffix on int literals 144 [19:52] <ClassBot> jsjgruber-l82-p asked: Any high precision float types in py3? 145 [19:53] <barry> jsjgruber-x-p_: i think so, but i do so little math in python, i can't help much with that 146 [19:53] <barry> i know there's a decimal type (though it's not built-in) 147 [19:54] <barry> the dict.iter*() methods are gone in py3. i.e. no more .iterkeys(), .itervalues(), or .iteritems() 148 [19:54] <barry> just use .keys() et al, and remember that they return iterators in py3 *not* concrete lists 149 [19:54] <barry> you might need to wrap the return value in list(dict.values()) for example 150 [19:55] <barry> the syntax for catching exceptions has changed (though also available in py2.6 and 2.7) 151 [19:55] <barry> gone is "except Foo, e" 152 [19:55] <barry> use 153 [19:55] <barry> "except Foo as e" instead 154 [19:55] <barry> instead of the exec statement, use the exec() function 155 [19:55] <ClassBot> There are 5 minutes remaining in the current session. 156 [19:55] <barry> file() builtin is gone, use open() 157 [19:56] <barry> no more xrange(). go back to range() which now returns an iterator 158 [19:56] <barry> as does filter() and map() 159 [19:56] <barry> (again, you might need to wrap them in list() to get a concrete list) 160 [19:56] <barry> absolute imports are the default, you must be explicit about relative imports 161 [19:57] <barry> okay, there's no way i'm going to finish. so are there any last minute questions? 162 [19:57] <barry> :) 163 [19:57] <barry> i'll just keep typing until classbot shuts me up 164 [19:57] <barry> 1 / 2 returns 0.5 now 165 [19:57] <barry> use 1//2 to get the old floor division behavior 166 [19:57] <ClassBot> coalitians asked: Barry, How and where should we get started to be part of ubuntu python community? 167 [19:58] <barry> coalitians: i would highly suggest joining the debian python community over on OFTC #debian-python. #ubuntu-devel is good too, as is #python on freenode 168 [19:58] <barry> there isn't a dedicated ubuntu python forum though. maybe there should be! 169 [19:59] <ClassBot> NickE68 asked: â€‹â€†Is there a place listing packages for which porting help is required (for 13.04 now, I guess)? 170 [19:59] <barry> NickE68: yes: 171 [19:59] <barry> https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AiT4gOXSkmapdFA1anRkWERsaXgtWnllUG9QWXhDVWc&pli=1#gid=0 172 [19:59] <barry> https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-python-versions 173 [19:59] <barry> https://wiki.ubuntu.com/Python/FoundationsQPythonVersions 174 [20:00] <barry> please ping me in #ubuntu-devel for more info. we'd *love* the help!