UsingAndPortingToPython3
Dev Week -- Using and porting to Python3 -- barry -- Tue, Aug 28th, 2012
1 [19:01] <barry> hello everyone. i'm barry warsaw and today i'll be giving a talk on python 3
2 [19:02] <barry> this is a broad topic, and there's way too much to cover in one hour so i'll probably be going fairly quickly to cover as much as possible
3 [19:02] <barry> it will really help if you at least already know pytho n2
4 [19:02] <barry> *python 2
5 [19:02] <barry> please join #ubuntu-classroom-chat and ask questions at any time!
6 [19:03] <barry> i'll be covering some of the new features in python 3, as well as porting strategies
7 [19:03] <barry> but i won't be talking much about porting c extensions or deb/ubu packaging (unless there's time)
8 [19:04] <barry> i probably won't be covering much of the py3 features that are already available in py2, except to the extent that it helps with explaining how best to port stuff
9 [19:04] <barry> i'll be providing lots of links with other resources you can read for much more detail
10 [19:05] <ClassBot> ben72 asked: ​ is there a way to run both python 2 and 3 in the same ubuntu system?
11 [19:06] <barry> ben72: yes. running 'python' (aka /usr/bin/python) on any ubuntu system will run the latest version of python 2, which for ubuntu 12.04 onward is python 2.7
12 [19:07] <barry> to run python 3 apps you need to explicitly run 'python3' or /usr/bin/python3
13 [19:07] <barry> same for #! lines
14 [19:07] <barry> i can talk about the difference between python 3.2, the default in ubuntu currently and python 3.3 later if you're interested
15 [19:08] <barry> hopefully, this will get you excited about python 3 -- it's fun!
16 [19:08] <ClassBot> NickE68 asked: ​ Will Quantal be using Python 3.3?
17 [19:09] <barry> NickE68: no, python 3.2 will be the default python 3 version because 3.3 was not ready in time for feature freeze. 3.3 will likely be the default in 13.04
18 [19:09] <barry> quantal does have 3.3 available though: apt-get install python3.3
19 [19:09] <barry> and invoked via 'python3.3'
20 [19:09] <barry> so you can play with the newest features easily
21 [19:10] <barry> also, i fully expect that 12.10 will get a ffe for the final release of 3.3 when it comes out
22 [19:10] <barry> so, to summarize: quantal will have 2.7, 3.2 and 3.3 with the former two as the defaults
23 [19:10] <barry> so, why port to python 3?
24 [19:11] <barry> there will never be a python 2.8. i can pretty much guarantee that :)
25 [19:11] <barry> which implies that py2 will never get any new features
26 [19:11] <barry> and diminishing bug fixes, though the official maintenance policy is longer than usual
27 [19:12] <barry> eventually 2.7 will only get security fixes, but i suspect upstream devs will become less interested in maintaining 2.7 as time goes on
28 [19:12] <barry> modules will rust
29 [19:12] <barry> py2 also has many warts, some big some small
30 [19:12] <barry> and, most importantly imo: py3 fixes the whole bytes vs text problem
31 [19:12] <barry> (mostly :)
32 [19:13] <barry> meaning, we have potential to make much more robust py3 apps for non-english locales
33 [19:13] <barry> i will talk more about bytes v. strings later
34 [19:13] <barry> what are ubuntu's py3 plans?
35 === Jameel is now known as Guest13536
36 [19:14] <barry> we are on a long journey to deprecate python2. we had hoped to port everything on the install images to py3 for 12.10, but we've hit a few big blockers so that won't happen in 12.10
37 [19:14] <barry> maybe for 13.04, definitely for the next lts
38 [19:15] <barry> py2.7 will never be *removed* from the archive, but it will likely only be in universe for 14.04 if we can get away with it
39 === Guest13536 is now known as JameelA
40 [19:16] <barry> if there are no questions related to ubuntu's py3 plans, let's talk about porting
41 [19:16] <barry> let me give you a few links first
42 [19:16] <barry> this is one of the very best resources available for understanding how to port your py2 code to py3:
43 [19:16] <barry> http://python3porting.com/
44 [19:17] <ClassBot> coalitians asked: So since the development for 12.10 is freezed, there are no ongoing porting activities now?
45 [19:17] <barry> coalitians: we will definitely have a ffe for the gwibber backend service py3 port. there may be a few others, but we're largely done with our porting efforts for 12.10. from here on, it will be bug fixing
46 [19:18] <barry> the above link talks about strategies, gives code samples for conversion of common idioms, a little bit about c extensions, and much more
47 [19:18] <barry> i have written a number of blog posts that you might find helpful:
48 [19:18] <barry> http://tinyurl.com/7tb3jkn
49 [19:19] <barry> http://tinyurl.com/7famvx3
50 [19:19] <barry> http://tinyurl.com/6ufpvfq
51 [19:19] <barry> there's also a good page in the ubuntu wiki with specific porting help, including how to package python libraries and apps either for py3 alone, or for dual py2/py3 versions
52 [19:20] <barry> https://wiki.ubuntu.com/Python/3
53 [19:20] <barry> i will also give you this link to a talk from pycon 2012. i highly recommend you watch this before you do any significant py3 work. it is the single best description of the bytes vs. strings issue i've seen. really a great talk:
54 [19:20] <barry> http://pyvideo.org/video/948/pragmatic-unicode-or-how-do-i-stop-the-pain
55 [19:21] <barry> okay, so let's talk for a moment about some porting strategies
56 [19:22] <barry> let me say first that if you are starting a brand new python project, i would suggest going straight to py3. one caveat is that you need to make sure any dependencies (i.e. 3rd party libraries you might want to use) are already available for py3, in ubuntu
57 [19:22] <barry> we've worked hard these last few cycles to make sure upstreams w/ py3 support have been packaged for deb and ubu
58 [19:22] <barry> and we've done some upstream porting work ourselves (e.g. dbus-python)
59 [19:23] <barry> of course, not everything is available, but *a lot* is, so you're probably in great shape with quantal
60 [19:23] <ClassBot> roadmr asked: what's the recommended path if I want to port an application using gstreamer to Python3?
61 [19:24] <barry> roadmr: i don't have a lot of experience w/gstreamer unfortunately, so i'm not sure if a py3 version is available, and if so, where (upstream, debian, ubuntu)
62 [19:24] <barry> i *think* it's not yet, but don't quote me on that :)
63 [19:25] <barry> so that's for new code, what about porting existing code to py3?
64 [19:25] <barry> ask this question: do you need to keep py2 support, and if so, how far back do you need to support?
65 [19:25] <barry> i would *highly* recommend nothing older than py2.6 (which actually is no longer available in quantal anyway)
66 [19:25] <barry> and would recommend at least py2.7 for some very useful features (e.g. the new unittest library)
67 [19:26] <barry> the thing is: py2.7 and to some extent 2.6 has lots of backported features that can make porting, and supporting py2 and py3 much easier
68 [19:26] <barry> through things like "from __future__ import"s
69 [19:27] <barry> so, when starting a porting effort, the first thing to do is to run your code with 'python2.7 -3'
70 [19:27] <barry> that -3 flag will cause python to warn you about any idiom that cannot be easily ported with the 2to3 program (more on that later)
71 [19:27] <barry> if your py2 program runs cleanly with 'python2.7 -3' then you're in pretty good shape for starting your py3 port
72 [19:28] <barry> 2to3 is a code transformation framework that can take most common py2 idioms and convert them on the fly to py3
73 [19:28] <barry> my personal opinion is that its output is useful but shouldn't be used "in production". it's a bit too slow in the dev cycle for me.
74 [19:28] <barry> but it's pretty flexible
75 [19:29] <barry> my own recommendation is to use a single code base and just be careful about how you code. use from-future imports as much as possible
76 [19:29] <barry> also there's a third party 'six' library (apt-get install python-six i believe) that can make a single code base easier to maintain
77 [19:30] <barry> but here's the most important thing you need to do before you port, and it segues into my next topic:
78 [19:30] <barry> you must be *crystal clear* about the distinction in your data model between bytes (i.e. binary data) and strings (i.e. human readable text)
79 [19:31] <barry> if you are getting lots of UnicodeErrors in your py2 code, it's a sign that you are not crystal clear :)
80 [19:31] <barry> and that is one of the biggest benefits (and frankly headaches while porting) of py3. the requirement to be clear means that once you've done so, your non-english users won't be getting UnicodeErrors
81 [19:32] <ClassBot> BlessJah asked: is it possible to write to write py3 code that would run also in py2 environment?
82 [19:32] <barry> BlessJah: that's not really much different than writing py2 code that runs in a py3 environment. yes, its very definitely possible
83 [19:33] <barry> so, hopefully everyone understand the difference between bytes and strings
84 [19:33] <barry> bytes are binary data, e.g. possibly images, audio, stuff like that
85 [19:33] <barry> strings are human readable text
86 [19:33] <barry> in py2 you had two "string" types, str which were also called 8-bit strings and they served as ascii text and binary data
87 [19:34] <barry> and unicodes which are just that, unicode human readable text
88 [19:34] <barry> the big py2 problem was that python auto-coerced between the two, often in ways that were not correct, leading to dreaded UnicodeErrors
89 [19:34] <barry> in py3 you have bytes which are *only* binary data
90 [19:34] <barry> and str which are unicodes
91 [19:35] <barry> and there is no auto-conversion
92 [19:35] <barry> you cannot concatenate the two without explicitly converting
93 [19:35] <barry> so if you have a unicode string, you can encode it to bytes, but you must know the encoding (e.g. utf-8 usually, but maybe latin-1 or any of the other multitude of encodings available)
94 [19:36] <barry> to go from bytes to str, you must decode, again by being explicit about the encoding
95 [19:36] <barry> an important thing to remember now too is that in py3, by default open() will open a file in binary mode and you read bytes out of it, *not* text
96 [19:37] <barry> unlike in py2, py3's open() takes an encoding argument, so if you call it like so: open('myfile', encoding='utf-8')
97 [19:37] <barry> you will be reading unicode strings from the file
98 [19:37] <barry> which is very nice (assuming of course that the contents of your file is utf-8 encoded strings :)
99 [19:38] <barry> in py3, there is no more u'' prefix for strings. unadorned string literals are unicodes
100 === keffie_jayx is now known as effiejayx
101 [19:38] <barry> (aside: the u'' prefix is reintroduced in py3.3 but essentially noops)
102 [19:38] <barry> to write bytes literals, you use the b'' prefix
103 [19:38] <barry> e.g. this is bytes: b'foo'
104 [19:38] <barry> and this is str: 'foo'
105 [19:39] <barry> one other quick note about this: you can now safely use the gettext module in py3 since it always returns unicodes from catalog lookups :)
106 [19:39] <barry> are there any questions about bytes vs strings before i move on?
107 [19:40] <ClassBot> NickE68 asked: ​ What if you don't know the encoding used for some file? Is there lib that can produce a 'best guess'?
108 [19:41] <barry> NickE68: there is, but i'm blanking on the name of it. i believe it comes out of mozilla. in general, it's much safer to know
109 [19:41] <barry> you can also try some of the most common ones, e.g. utf-8 first, then latin-1
110 [19:41] <barry> then ascii
111 [19:41] <ClassBot> roadmr asked: gettext, "it always returns unicodes". Do you mean "strings" here, as opposed to "bytes"?
112 [19:42] <barry> roadmr: yes. in py2 unfortunately, gettext.gettext('foo') always returned an 8-bit string, which really causes problems, and the api for getting (py2) unicodes is pretty well hidden and not the default
113 [19:42] <barry> in py3, gettext.gettext('foo') always returns a str, aka a unicode
114 [19:43] <barry> jtaylor: reminds me that it's python-chardet. thanks!
115 [19:43] <barry> cool. so, in the remaining 17m or so, i'm going to give a whirlwind tour of some of the changes and new features in py3 to be aware of
116 [19:43] <barry> this is by no means exhaustive!
117 [19:44] <barry> there are "what's new" documents out there for each individual version of python 3, but afaik, no overarching list of differences b/w say py2.7 and py3.2
118 [19:44] <barry> though the pythonporting url i gave above comes pretty close
119 [19:45] <barry> so, let's begin, and please ask about further details at any point
120 [19:45] <barry> many parts of the stdlib have been reorganized
121 [19:45] <barry> this means that some old, unsupported modules have been removed
122 [19:46] <barry> probably nothing you care about unless you're doing image processing on irix and then why are you here? :)
123 [19:46] <barry> many modules have been moved around, some placed inside of packages
124 [19:46] <barry> e.g. the very common 'urlparse()' function now lives in urllib.parse.urlparse()
125 [19:47] <barry> http://docs.python.org/py3k/library/ is your best friend
126 [19:47] <barry> some builtins have been removed or moved
127 [19:47] <barry> e.g. the reload() builtin has been moved to imp.reload()
128 [19:48] <barry> for the builtins that have been removed, there's usually a good alternative
129 [19:48] <barry> e.g. callable() is gone, but you can test for callability either by just calling it <wink> or through the inspect module
130 [19:49] <barry> one big change that gave me much finger pain until i re-taught the muscle memory (and started using "from __future__ import print_function" in my py2 code)
131 [19:49] <barry> is that the print statement has become a function
132 [19:49] <barry> e.g. print('hello world')
133 [19:49] <barry> this was very jarring, but it makes great sense, and eventually you'll come to appreciate this change :)
134 [19:50] <barry> you know how python2 has essentially two types of classes? classic classes and "new style" classes?
135 [19:50] <ClassBot> There are 10 minutes remaining in the current session.
136 [19:50] <barry> py3 gets rid of classic classes. all classes in py3 are "new style"
137 [19:50] <barry> so that means you no longer have to inherit from object
138 [19:51] <barry> or use the "__metaclass__ = type" trick to get all the great features of "new style" classes
139 [19:51] <barry> i guess that means we have to stop calling them "new style" :)
140 [19:51] <barry> py2 has two types of integer, int and long
141 [19:51] <barry> py3 does away with this and has only one integer type, int
142 [19:52] <barry> it automatically knows to use the more efficient internal representation for smaller ints and the expensive representation for bigger ints
143 [19:52] <barry> there is no more 'L' suffix on int literals
144 [19:52] <ClassBot> jsjgruber-l82-p asked: Any high precision float types in py3?
145 [19:53] <barry> jsjgruber-x-p_: i think so, but i do so little math in python, i can't help much with that
146 [19:53] <barry> i know there's a decimal type (though it's not built-in)
147 [19:54] <barry> the dict.iter*() methods are gone in py3. i.e. no more .iterkeys(), .itervalues(), or .iteritems()
148 [19:54] <barry> just use .keys() et al, and remember that they return iterators in py3 *not* concrete lists
149 [19:54] <barry> you might need to wrap the return value in list(dict.values()) for example
150 [19:55] <barry> the syntax for catching exceptions has changed (though also available in py2.6 and 2.7)
151 [19:55] <barry> gone is "except Foo, e"
152 [19:55] <barry> use
153 [19:55] <barry> "except Foo as e" instead
154 [19:55] <barry> instead of the exec statement, use the exec() function
155 [19:55] <ClassBot> There are 5 minutes remaining in the current session.
156 [19:55] <barry> file() builtin is gone, use open()
157 [19:56] <barry> no more xrange(). go back to range() which now returns an iterator
158 [19:56] <barry> as does filter() and map()
159 [19:56] <barry> (again, you might need to wrap them in list() to get a concrete list)
160 [19:56] <barry> absolute imports are the default, you must be explicit about relative imports
161 [19:57] <barry> okay, there's no way i'm going to finish. so are there any last minute questions?
162 [19:57] <barry> :)
163 [19:57] <barry> i'll just keep typing until classbot shuts me up
164 [19:57] <barry> 1 / 2 returns 0.5 now
165 [19:57] <barry> use 1//2 to get the old floor division behavior
166 [19:57] <ClassBot> coalitians asked: Barry, How and where should we get started to be part of ubuntu python community?
167 [19:58] <barry> coalitians: i would highly suggest joining the debian python community over on OFTC #debian-python. #ubuntu-devel is good too, as is #python on freenode
168 [19:58] <barry> there isn't a dedicated ubuntu python forum though. maybe there should be!
169 [19:59] <ClassBot> NickE68 asked: ​ Is there a place listing packages for which porting help is required (for 13.04 now, I guess)?
170 [19:59] <barry> NickE68: yes:
171 [19:59] <barry> https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AiT4gOXSkmapdFA1anRkWERsaXgtWnllUG9QWXhDVWc&pli=1#gid=0
172 [19:59] <barry> https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-python-versions
173 [19:59] <barry> https://wiki.ubuntu.com/Python/FoundationsQPythonVersions
174 [20:00] <barry> please ping me in #ubuntu-devel for more info. we'd *love* the help!
MeetingLogs/devweek1208/UsingAndPortingToPython3 (last edited 2012-08-29 10:46:01 by dholbach)