1 [17:59] <cjwatson> I'll hand over now to Jonathan Lange, a gentleman and a scholar, who's going to be talking about launchpadlib.
   2 [17:59] <jml> cjwatson, why thank you
   3 [18:00] <jml> as before, please stop me if I go too fast, or hassle me if I go too slow
   4 [18:00] <jml> You might know already, but pretty much no matter what kind of Ubuntu development you are doing, you'll have to interact with Launchpad
   5 [18:00] <jml> Interacting with Launchpad is fun and all, but sometimes you'll want a computer to do the interaction for you
   6 [18:00] <jml> Say you want to suck out some of the data and make a pretty graph
   7 [18:00] <jml> or you want to build a site like http://qa.ubuntu.com/
   8 [18:00] <jml> This session should help you do that...
   9 [18:00] <jml> Before we begin, you should install the 'python-launchpadlib' package now.
  10 [18:01] <jml> Confirm that it works by running:
  11 [18:01] <jml> $ python
  12 [18:01] <jml> and then
  13 [18:01] <jml> >>> import launchpadlib
  14 [18:01] <jml> >>> launchpadlib.__version__
  15 [18:01] <jml> you should get output like:
  16 [18:01] <jml> '1.5.1'
  17 [18:01] <jml> if it's an earlier version, then you need to either update the package, upgrade to a newer version of Ubuntu, or find the PPA.
  18 [18:01] <jml> A moment for those installing the package
  19 [18:01] <jml> Moment over.
  20 [18:01] <jml> This session is going to have four parts: theory, example, gotchas and future.
  21 [18:02] <jml> = 1. Theory =
  22 [18:02] <jml> The Launchpad developers implemented an API for three reasons:
  23 [18:02] <jml> 1. To let you get at your data
  24 [18:02] <jml> 2. To allow you to do project-specific policy stuff
  25 [18:02] <jml> 3. To watch people do cool things we would never have thought of.
  26 [18:02] <jml> (I'm a Launchpad developer, hence the 'we')
  27 [18:02] <jml> We decided to implement the API using something called RESTful Architecture
  28 [18:02] <jml> http://en.wikipedia.org/wiki/Representational_State_Transfer
  29 [18:03] <jml> This means that Launchpad's APIs are actually language neutral
  30 [18:03] <jml> in theory, you can write code to control Launchpad in any language you want
  31 [18:03] <jml> without screenscraping!
  32 [18:03] <jml> You do HTTP requests with operations like GET, POST, PUSH and DELETE and send and receive JSON content that's described by a WADL file
  33 [18:03] <jml> WADL == http://en.wikipedia.org/wiki/Web_Application_Description_Language
  34 [18:03] <jml> Which is all nice and wonderful but you don't really have to care about it, because there's also a Python library
  35 [18:03] <jml> called "launchpadlib"
  36 [18:04] <jml> (have you installed it yet?)
  37 [18:04] <jml> If you don't care at all about Python, now's the time to stop following this session and go to https://help.launchpad.net/API instead
  38 [18:04] <jml> Because I'm about to go into the example.
  39 [18:04] <jml> = 2. Example =
  40 [18:04] <jml> You can follow along from this example by getting the branch at lp:~jml/+junk/bugstats
  41 [18:04] <jml> You do that by running 'bzr branch lp:~jml/+junk/bugstats'
  42 [18:04] <jml> The way to follow along is to change into that branch, and then run 'bzr revert -r 1'
  43 [18:04] <jml> and then when I say "Next revision", do the same thing with "-r 2" instead
  44 [18:04] <jml> and so on
  45 [18:05] <jml> bzr revert -r1
  46 [18:05] <jml> The branch contains a single file 'bugstats.py'
  47 [18:05] <jml> If you look at that file in your favorite text editor, you'll see that it's very very simple
  48 [18:05] <jml> It imports sys, defines a main function that does nothing, and then runs that main function.
  49 [18:05] <jml> Go ahead and run it now
  50 [18:05] <jml> $ python bugstats.py
  51 [18:05] <jml> It should do absolutely nothing, fairly quickly
  52 [18:05] <jml> (at the speed of PYTHON!)
  53 [18:05] <jml> OK. Next revision
  54 [18:06] <jml> bzr revert -r2
  55 [18:06] <jml> (sorry alt-tab fail)
  56 [18:06] <jml> Have a look at this version of bugstats.py. You might need to reload it in your editor.
  57 [18:06] <jml> This is an extremely simple launchpadlib script
  58 [18:06] <jml> It logs in to launchpad and prints out the title of bug number one
  59 [18:06] <jml> I'll break down exactly what the script does in a moment.
  60 [18:06] <jml> But first, you should run the script.
  61 [18:06] <jml> $ python bugstats.py
  62 [18:06] <jml> When you run it, an authorization page will open up in your web browser
  63 [18:07] <jml> This is very important, this is how your Launchpad web credentials get applied to the application
  64 [18:07] <jml> It's done using oauth, which I don't understand.
  65 [18:07] <jml> http://oauth.net/ and http://en.wikipedia.org/wiki/OAuth are probably good places to start though.
  66 [18:07] <jml> For now, pick "Read non-private data"
  67 [18:07] <jml> Then, switch back to your terminal and press enter.
  68 [18:07] <jml> The script should then print the title of bug number one.
  69 [18:07] <jml> Run the script again
  70 [18:07] <jml> Notice that launchpadlib has cached your credentials
  71 [18:08] <jml>     launchpad = Launchpad.login_with(APP_NAME, SERVICE_ROOT, CACHE_DIR)
  72 [18:08] <jml> ^^^ that's the line of code what did it
  73 [18:08] <jml> if you look at the script...
  74 [18:08] <jml> you'll see I've imported 'os' and 'sys', which are standard Python modules.
  75 [18:08] <jml> I've also imported 'Launchpad' and 'STAGING_SERVICE_ROOT' from launchpadlib
  76 [18:08] <jml> Then I've defined an application name, a caching directory and a service root
  77 [18:08] <jml> APP_NAME = 'jml-bug-stats'
  78 [18:08] <jml> CACHE_DIR = os.path.expanduser('~/.launchpadlib/cache')
  79 [18:08] <jml> SERVICE_ROOT = STAGING_SERVICE_ROOT
  80 [18:08] <jml> There's nothing special about the names of the variables, I've just made them ALL CAPS to hint that they are constants
  81 [18:08] <jml> APP_NAME is the name of the application. launchpadlib & Launchpad use this as a marker for authorization. You authorize an application based on its *name*.
  82 [18:09] <jml> Go to https://staging.launchpad.net/people/+me/+oauth-tokens and search for "jml-bug-stats"
  83 [18:09] <jml> you'll only find it if you actually ran the script
  84 [18:09] <jml> if you have run the script, you should find it there, saying when you authorized it and what you authorized it to do
  85 [18:09] <jml> You could revoke the authorization if you wanted to, but you shouldn't do that just now
  86 [18:09] <jml> Anyway, that 'jml-bug-stats' is the app name used in the script.
  87 [18:10] <jml> SERVICE_ROOT is _which_ Launchpad.net instance to use
  88 [18:10] <jml> does everyone know about staging.launchpad.net, edge.launchpad.net and so forth?
  89 [18:10] <jml> ...
  90 [18:10] <jml> that's a yes.
  91 [18:10] <jml> Today, we're using staging. Testing against staging is a good idea, since it's impossible to mess up your live production data by messing with staging.
  92 [18:10] <jml> CACHE_DIR is the place where launchpadlib stores cached credentials and cached json objects
  93 [18:11] <jml> It's worth having a poke around in it sometime
  94 [18:11] <jml> most apps use ~/.launchpadlib/cache
  95 [18:11] <jml> for reasons that escape me
  96 [18:11] <jml> If you run ls -l ~/.launchpadlib/cache/api.staging.launchpad.net/credentials/ you should see a file called 'jml-bug-stats'
  97 [18:11] <jml> That file contains the cached credentials for this app
  98 [18:11] <jml> If you delete it, you'll have to re-authorize the application
  99 [18:11] <jml> moving on to the script proper.
 100 [18:11] <jml> The main() function has us logging into Launchpad and printing a bug title
 101 [18:11] <jml> login_with is a handy convenience function that logs in with an application name, but looks in the cached credentials first to see if you've already logged in. It returns an instance of 'Launchpad'
 102 [18:11] <jml> 'launchpad' is an object with a bunch of pre-defined top-level collections, and is the starting point of any launchpadlib app.
 103 [18:12] <jml> https://help.launchpad.net/API/launchpadlib#The%20top-level%20objects will list all of them for you
 104 [18:12] <jml> Here we use 'bugs', get the bug with id '1' and print its title.
 105 [18:12] <jml> $ python bugstats.py
 106 [18:12] <jml> Microsoft has a majority market share
 107 [18:12] <jml> Easy
 108 [18:12] <jml> Now for something a bit more useful
 109 [18:12] <jml> Next revision
 110 [18:12] <jml> bzr revert -r3
 111 [18:12] <jml> $ python bugstats.py
 112 [18:12] <jml> Please specify a project and a username.
 113 [18:13] <jml> $ python bugstats.py ubuntu jml
 114 [18:13] <jml> Jonathan Lange has had 2 bugs fixed on Ubuntu
 115 [18:13] <jml> it shows the number of bugs someone has had fixed
 116 [18:13] <jml> useful, huh?
 117 [18:13] <jml> take a look at the code
 118 [18:13] <jml> It's got a work-around for a bug in launchpadlib to get the length of a returned collection.
 119 [18:13] <jml> I don't really understand why the bug isn't fixed yet, since it's got to bite almost everyone who writes a non-trivial launchpadlib application.
 120 [18:13] <jml> Anyway, this script is more advanced than the last one.
 121 [18:14] <jml> look at ...
 122 [18:14] <jml>     pillar = launchpad.projects[pillar_name]
 123 [18:14] <jml> "Pillar" is Launchpad developer jargon for a distribution, project or project group. e.g. ubuntu, gnome-do or gnome
 124 [18:14] <jml> you can try running the script with your username and fave project
 125 [18:14] <jml> The script takes the pillar name from the command line, along with a bug reporter name, finds all of the 'fix released' bugs and then prints out a friendly message.
 126 [18:14] <jml>     reporter = launchpad.people[reporter_name]
 127 [18:14] <jml>     fixed_bugtasks = pillar.searchTasks(
 128 [18:14] <jml>         bug_reporter=reporter, status=['Fix Released'])
 129 [18:15] <jml> If I run it with this:
 130 [18:15] <jml> $ python bugstats.py launchpad-code jml
 131 [18:15] <jml> I get:
 132 [18:15] <jml> Jonathan Lange has had 164 bugs fixed on Launchpad Bazaar Integration
 133 [18:15] <jml> To me, the most interesting thing about this script (apart from the cool data) is "how did I figure out to call 'searchTasks'?"
 134 [18:15] <jml> Well, each of the Launchpad instances has generated API documentation at +apidoc
 135 [18:15] <jml> e.g. https://staging.launchpad.net/+apidoc/
 136 [18:15] <jml> or e.g. https://edge.launchpad.net/+apidoc/
 137 [18:15] <jml> I happen to know a bit about the Launchpad object model, and knew I'd need to get a bunch of "bug tasks"
 138 [18:16] <jml> A bug task is that row in the table that has an assignee, status, importance etc. A bug can have many bug tasks.
 139 [18:16] <jml> I went to the API documentation page and had a look around for something that would return a list of bug tasks.
 140 [18:16] <jml> I found "searchTasks", and then gave it a try and it worked
 141 [18:16] <jml> The API documentation also told me that people have display_name attributes and that projects do too.
 142 [18:16] <jml> Note that the API documentation is *not* launchpadlib documentation. It's generic REST API documentation.
 143 [18:16] <jml> That means you often need to translate from the abstract REST docs into concrete Python. It's not that hard.
 144 [18:16] <jml> Anyway the script is pretty simple
 145 [18:16] <jml> <geser> QUESTION: does python-launchpadlib also clean up the cache of old unused files?
 146 [18:16] <jml> geser, No, it does not.
 147 [18:17] <jml> geser, as far as I'm aware, there's absolutely no functionality in launchpadlib for removing cache files under any circumstances
 148 [18:17] <jml> The next revision is something a little bit more complex
 149 [18:17] <jml> bzr revert -r4
 150 [18:17] <jml> This script does almost exactly the same thing, except it tells you how successful the reporter was as a percentage of bugs fixed over bugs filed
 151 [18:18] <jml> The new code is the total_bugtasks line, which finds all bugs of any status
 152 [18:18] <jml>     fixed_bugtasks = pillar.searchTasks(
 153 [18:18] <jml>         bug_reporter=reporter, status=['Fix Released'])
 154 [18:18] <jml>     total_bugtasks = pillar.searchTasks(
 155 [18:18] <jml>         bug_reporter=reporter,
 156 [18:18] <jml>         status=[
 157 [18:18] <jml>             "New",
 158 [18:18] <jml>             "Incomplete",
 159 [18:18] <jml>             "Invalid",
 160 [18:18] <jml>             "Won't Fix",
 161 [18:18] <jml>             "Confirmed",
 162 [18:18] <jml>             "Triaged",
 163 [18:18] <jml>             "In Progress",
 164 [18:18] <jml>             "Fix Committed",
 165 [18:18] <jml>             'Fix Released'])
 166 [18:18] <jml> zoiks!
 167 [18:18] <jml> (I thought I had my newlines done differently)
 168 [18:19] <jml> I had to specify each status manually, because 'searchTasks' defaults to only finding open bugs
 169 [18:19] <jml> Then there's some code to calculate a percentage and print it out nicely
 170 [18:19] <jml>     percentage = 100.0 * length(fixed_bugtasks) / length(total_bugtasks)
 171 [18:19] <jml>     print "%s is %.2f%% successful on bugs in %s" % (
 172 [18:19] <jml>         reporter.display_name, percentage, pillar.display_name)
 173 [18:19] <jml> Let's try running it!
 174 [18:19] <jml> $ python bugstats.py ubuntu jml
 175 [18:19] <jml> Jonathan Lange is 22.22% successful on bugs in Ubuntu
 176 [18:19] <jml> $ python bugstats.py launchpad-code jml
 177 [18:19] <jml> Jonathan Lange is 56.16% successful on bugs in Launchpad Bazaar Integration
 178 [18:20] <jml> Again, you can try it with yourself, or with a friend, or with cjwatson
 179 [18:20] <jml> That's all I have for the example.
 180 [18:20] <jml> = 3. Gotchas =
 181 [18:20] <jml> There are quite a few things that can trip you up with the Launchpad API
 182 [18:20] <jml> We've already bumped into a couple of them: it's got bugs.
 183 [18:20] <jml> https://bugs.edge.launchpad.net/launchpadlib/ for more info on that
 184 [18:21] <jml> The second is that even though it does have documentation, it's not always easy to apply to what you need.
 185 [18:21] <jml> (wtf, "Colin Watson is 79.65% successful on bugs in Ubuntu"!)
 186 [18:21] <jml> I'm going to pimp http://help.launchpad.net/API again, since there really is a fair chunk of good documentation there.
 187 [18:21] <jml> https://help.launchpad.net/API/Examples is good, as is https://help.launchpad.net/API/launchpadlib
 188 [18:21] <jml> However, the docs often aren't up to scratch.
 189 [18:22] <jml> There is only one way for them to get better
 190 [18:22] <jml> People like your good selves must ask questions, get answers, and then edit the help.launchpad.net wiki.
 191 [18:22] <jml> Another gotcha is error messages.
 192 [18:22] <jml> Because launchpadlib is a very, very thin wrapper over a generic RESTful client library, if ever you get an exception raised, it's going to be really unhelpful.
 193 [18:22] <jml> https://help.launchpad.net/API/Examples#Get%20a%20useful%20error%20message%20from%20launchpadlib shows the work-around.
 194 [18:22] <jml> I think I filed a bug about that.
 195 [18:23] <jml> hmm.
 196 [18:23] <jml> there's another gotcha, which is *performance*
 197 [18:23] <jml> For a lot of the things you probably want to do, you'll end up writing code that looks like this:
 198 [18:23] <jml>   for thing in bunch_of_things:
 199 [18:23] <jml>       thing.do_something_on_launchpad()
 200 [18:23] <jml> Code like this is really really slow
 201 [18:23] <jml> It'll do an HTTPS request for each 'thing'
 202 [18:23] <jml> We'd like to have some way of reducing that load, but we don't have any good answers yet
 203 [18:23] <jml> Only two more gotchas left :)
 204 [18:23] <jml> Not everything that's in Launchpad itself is exposed via the API.
 205 [18:23] <jml> Sorry.
 206 [18:23] <jml> It's generally either really, really easy to expose stuff over the API or almost impossible.
 207 [18:24] <jml> If the thing you want falls into the first category, you can probably patch Launchpad yourself.
 208 [18:24] <jml> In any case, *file a bug*. Everything that's not exposed over the API is a bug.
 209 [18:24] <jml> Last gotcha: testing launchpadlib apps is hard
 210 [18:24] <jml> jkakar has done some work on this recently
 211 [18:24] <jml> but I haven't had a chance to look at it.
 212 [18:24] <jml> That's it on the gotchas: bugs, docs, errors, potato programming, unexposed methods, testing
 213 [18:25] <jml> = 4. Future =
 214 [18:25] <jml> There are plenty of other examples of launchpadlib apps out there
 215 [18:25] <jml> bughugger uses launchpadlib (https://launchpad.net/bughugger)
 216 [18:25] <jml> quickly has some templates for using launchpadlib (https://launchpad.net/quickly)
 217 [18:25] <jml> The code that puts branches in https://code.launchpad.net/ubuntu is done entirely with launchpadlib (we talked about those branches last session!)
 218 [18:25] <jml> There's also a stack of projects that are part of a group called 'lpx' that you can find at https://launchpad.net/lpx
 219 [18:25] <jml> and at http://help.launchpad.net/API/Uses there's even more stuff
 220 [18:25] <jml> If ever you need help, ask on #launchpad or #launchpad-dev on freenode or send an email to launchpad-users@lists.launchpad.net
 221 [18:26] <jml> That's it from me.
 222 [18:26] <jml> Questions?
 223 [18:26] <jml> Cool ideas for launchpadlib programs?
 224 [18:26] <jml> General complaints about Launchpad that I can respond soothingly to?
 225 [18:27] <jml> By "complaint", I of course meant "suggestion for improvement"
 226 [18:27] <jml> <vocx> QUESTION: so, Python plays a HUGE part in Ubuntu, no? Other scripting won't feel left out?
 227 [18:27] <jml> A lot of Ubuntu stuff is written in Python.
 228 [18:28] <jml> As I mentioned earlier, you don't *have* to use Python to control Launchpad, but it makes it a lot easier.
 229 [18:28] <jml> <jönöbacon>  QUESTION: Can you make more examples for python-snippets happen?
 230 [18:29] <jml> jonobacon, I can point you at https://help.launchpad.net/API/Examples again, I guess :)
 231 [18:29] <jml> <kamalmostafa> QUESTION: Is there a way to access two JSON values in one "call"... I.e.   launchpad.people[reporter_name].latitude   gets the latitude -- is there a way to also get the longitude at the same time?
 232 [18:30] <jml> kamalmostafa, I'm fairly sure that if you get the object first, then subsequent attribute access doesn't do a separate webapp call.
 233 [18:30] <jml> so...
 234 [18:30] <jml> reporter = launchpad.people[reporter_name]
 235 [18:30] <jml> reporter.latitude, reporter.longtitude
 236 [18:30] <jml> <vocx> QUESTION: is there a need to worry about python 3.0 breaking something?
 237 [18:30] <jml> vocx, wrt launchpadlib or wrt Ubuntu more generally?
 238 [18:31] <jml> vocx, launchpadlib is a Python 2 application. It hasn't been tested with Python 3.
 239 [18:32] <jml> vocx, I'm not the launchpadlib maintainer, but I tend to think of python3 as a completely different language. if a program works on both 2 and 3 it's a happy accident
 240 [18:33] <jml> vocx, no plans to port to 3 yet.
 241 [18:33] <jml> vocx, I guess we'd start doing that when Ubuntu stops providing Python 2
 242 [18:34] <jml> are there any other questions?
 243 [18:34] <jml> QUESTION: Can we convince jonobacon to use metal umlauts in his name so my friends can go back to calling me Jono?
 244 === jml is now known as joneaux
 245 === joneaux is now known as jml
 246 [18:37] <jml> Last chance for questions folks
 247 [18:37] <jml> remember you can get the code for the example by 'bzr branch lp:~jml/+junk/bugstats'
 248 [18:37] <jml> and that the folk on #launchpad-dev are helpful and friendly and want more people doing stuff with launchpadlib
 249 [18:38] <jml> ok folks