## page was renamed from MeetingLogs/devweek1001/devweek1001/LaunchpadLib == Dev Week -- Meet launchpadlib -- jml -- Fri, Jan 29 == UTC {{{#!IRC [17:59] I'll hand over now to Jonathan Lange, a gentleman and a scholar, who's going to be talking about launchpadlib. [17:59] cjwatson, why thank you [18:00] as before, please stop me if I go too fast, or hassle me if I go too slow [18:00] You might know already, but pretty much no matter what kind of Ubuntu development you are doing, you'll have to interact with Launchpad [18:00] Interacting with Launchpad is fun and all, but sometimes you'll want a computer to do the interaction for you [18:00] Say you want to suck out some of the data and make a pretty graph [18:00] or you want to build a site like http://qa.ubuntu.com/ [18:00] This session should help you do that... [18:00] Before we begin, you should install the 'python-launchpadlib' package now. [18:01] Confirm that it works by running: [18:01] $ python [18:01] and then [18:01] >>> import launchpadlib [18:01] >>> launchpadlib.__version__ [18:01] you should get output like: [18:01] '1.5.1' [18:01] if it's an earlier version, then you need to either update the package, upgrade to a newer version of Ubuntu, or find the PPA. [18:01] A moment for those installing the package [18:01] Moment over. [18:01] This session is going to have four parts: theory, example, gotchas and future. [18:02] = 1. Theory = [18:02] The Launchpad developers implemented an API for three reasons: [18:02] 1. To let you get at your data [18:02] 2. To allow you to do project-specific policy stuff [18:02] 3. To watch people do cool things we would never have thought of. [18:02] (I'm a Launchpad developer, hence the 'we') [18:02] We decided to implement the API using something called RESTful Architecture [18:02] http://en.wikipedia.org/wiki/Representational_State_Transfer [18:03] This means that Launchpad's APIs are actually language neutral [18:03] in theory, you can write code to control Launchpad in any language you want [18:03] without screenscraping! [18:03] You do HTTP requests with operations like GET, POST, PUSH and DELETE and send and receive JSON content that's described by a WADL file [18:03] WADL == http://en.wikipedia.org/wiki/Web_Application_Description_Language [18:03] Which is all nice and wonderful but you don't really have to care about it, because there's also a Python library [18:03] called "launchpadlib" [18:04] (have you installed it yet?) [18:04] If you don't care at all about Python, now's the time to stop following this session and go to https://help.launchpad.net/API instead [18:04] Because I'm about to go into the example. [18:04] = 2. Example = [18:04] You can follow along from this example by getting the branch at lp:~jml/+junk/bugstats [18:04] You do that by running 'bzr branch lp:~jml/+junk/bugstats' [18:04] The way to follow along is to change into that branch, and then run 'bzr revert -r 1' [18:04] and then when I say "Next revision", do the same thing with "-r 2" instead [18:04] and so on [18:05] bzr revert -r1 [18:05] The branch contains a single file 'bugstats.py' [18:05] If you look at that file in your favorite text editor, you'll see that it's very very simple [18:05] It imports sys, defines a main function that does nothing, and then runs that main function. [18:05] Go ahead and run it now [18:05] $ python bugstats.py [18:05] It should do absolutely nothing, fairly quickly [18:05] (at the speed of PYTHON!) [18:05] OK. Next revision [18:06] bzr revert -r2 [18:06] (sorry alt-tab fail) [18:06] Have a look at this version of bugstats.py. You might need to reload it in your editor. [18:06] This is an extremely simple launchpadlib script [18:06] It logs in to launchpad and prints out the title of bug number one [18:06] I'll break down exactly what the script does in a moment. [18:06] But first, you should run the script. [18:06] $ python bugstats.py [18:06] When you run it, an authorization page will open up in your web browser [18:07] This is very important, this is how your Launchpad web credentials get applied to the application [18:07] It's done using oauth, which I don't understand. [18:07] http://oauth.net/ and http://en.wikipedia.org/wiki/OAuth are probably good places to start though. [18:07] For now, pick "Read non-private data" [18:07] Then, switch back to your terminal and press enter. [18:07] The script should then print the title of bug number one. [18:07] Run the script again [18:07] Notice that launchpadlib has cached your credentials [18:08] launchpad = Launchpad.login_with(APP_NAME, SERVICE_ROOT, CACHE_DIR) [18:08] ^^^ that's the line of code what did it [18:08] if you look at the script... [18:08] you'll see I've imported 'os' and 'sys', which are standard Python modules. [18:08] I've also imported 'Launchpad' and 'STAGING_SERVICE_ROOT' from launchpadlib [18:08] Then I've defined an application name, a caching directory and a service root [18:08] APP_NAME = 'jml-bug-stats' [18:08] CACHE_DIR = os.path.expanduser('~/.launchpadlib/cache') [18:08] SERVICE_ROOT = STAGING_SERVICE_ROOT [18:08] There's nothing special about the names of the variables, I've just made them ALL CAPS to hint that they are constants [18:08] APP_NAME is the name of the application. launchpadlib & Launchpad use this as a marker for authorization. You authorize an application based on its *name*. [18:09] Go to https://staging.launchpad.net/people/+me/+oauth-tokens and search for "jml-bug-stats" [18:09] you'll only find it if you actually ran the script [18:09] if you have run the script, you should find it there, saying when you authorized it and what you authorized it to do [18:09] You could revoke the authorization if you wanted to, but you shouldn't do that just now [18:09] Anyway, that 'jml-bug-stats' is the app name used in the script. [18:10] SERVICE_ROOT is _which_ Launchpad.net instance to use [18:10] does everyone know about staging.launchpad.net, edge.launchpad.net and so forth? [18:10] ... [18:10] that's a yes. [18:10] Today, we're using staging. Testing against staging is a good idea, since it's impossible to mess up your live production data by messing with staging. [18:10] CACHE_DIR is the place where launchpadlib stores cached credentials and cached json objects [18:11] It's worth having a poke around in it sometime [18:11] most apps use ~/.launchpadlib/cache [18:11] for reasons that escape me [18:11] If you run ls -l ~/.launchpadlib/cache/api.staging.launchpad.net/credentials/ you should see a file called 'jml-bug-stats' [18:11] That file contains the cached credentials for this app [18:11] If you delete it, you'll have to re-authorize the application [18:11] moving on to the script proper. [18:11] The main() function has us logging into Launchpad and printing a bug title [18:11] login_with is a handy convenience function that logs in with an application name, but looks in the cached credentials first to see if you've already logged in. It returns an instance of 'Launchpad' [18:11] 'launchpad' is an object with a bunch of pre-defined top-level collections, and is the starting point of any launchpadlib app. [18:12] https://help.launchpad.net/API/launchpadlib#The%20top-level%20objects will list all of them for you [18:12] Here we use 'bugs', get the bug with id '1' and print its title. [18:12] $ python bugstats.py [18:12] Microsoft has a majority market share [18:12] Easy [18:12] Now for something a bit more useful [18:12] Next revision [18:12] bzr revert -r3 [18:12] $ python bugstats.py [18:12] Please specify a project and a username. [18:13] $ python bugstats.py ubuntu jml [18:13] Jonathan Lange has had 2 bugs fixed on Ubuntu [18:13] it shows the number of bugs someone has had fixed [18:13] useful, huh? [18:13] take a look at the code [18:13] It's got a work-around for a bug in launchpadlib to get the length of a returned collection. [18:13] I don't really understand why the bug isn't fixed yet, since it's got to bite almost everyone who writes a non-trivial launchpadlib application. [18:13] Anyway, this script is more advanced than the last one. [18:14] look at ... [18:14] pillar = launchpad.projects[pillar_name] [18:14] "Pillar" is Launchpad developer jargon for a distribution, project or project group. e.g. ubuntu, gnome-do or gnome [18:14] you can try running the script with your username and fave project [18:14] The script takes the pillar name from the command line, along with a bug reporter name, finds all of the 'fix released' bugs and then prints out a friendly message. [18:14] reporter = launchpad.people[reporter_name] [18:14] fixed_bugtasks = pillar.searchTasks( [18:14] bug_reporter=reporter, status=['Fix Released']) [18:15] If I run it with this: [18:15] $ python bugstats.py launchpad-code jml [18:15] I get: [18:15] Jonathan Lange has had 164 bugs fixed on Launchpad Bazaar Integration [18:15] To me, the most interesting thing about this script (apart from the cool data) is "how did I figure out to call 'searchTasks'?" [18:15] Well, each of the Launchpad instances has generated API documentation at +apidoc [18:15] e.g. https://staging.launchpad.net/+apidoc/ [18:15] or e.g. https://edge.launchpad.net/+apidoc/ [18:15] I happen to know a bit about the Launchpad object model, and knew I'd need to get a bunch of "bug tasks" [18:16] A bug task is that row in the table that has an assignee, status, importance etc. A bug can have many bug tasks. [18:16] I went to the API documentation page and had a look around for something that would return a list of bug tasks. [18:16] I found "searchTasks", and then gave it a try and it worked [18:16] The API documentation also told me that people have display_name attributes and that projects do too. [18:16] Note that the API documentation is *not* launchpadlib documentation. It's generic REST API documentation. [18:16] That means you often need to translate from the abstract REST docs into concrete Python. It's not that hard. [18:16] Anyway the script is pretty simple [18:16] QUESTION: does python-launchpadlib also clean up the cache of old unused files? [18:16] geser, No, it does not. [18:17] geser, as far as I'm aware, there's absolutely no functionality in launchpadlib for removing cache files under any circumstances [18:17] The next revision is something a little bit more complex [18:17] bzr revert -r4 [18:17] This script does almost exactly the same thing, except it tells you how successful the reporter was as a percentage of bugs fixed over bugs filed [18:18] The new code is the total_bugtasks line, which finds all bugs of any status [18:18] fixed_bugtasks = pillar.searchTasks( [18:18] bug_reporter=reporter, status=['Fix Released']) [18:18] total_bugtasks = pillar.searchTasks( [18:18] bug_reporter=reporter, [18:18] status=[ [18:18] "New", [18:18] "Incomplete", [18:18] "Invalid", [18:18] "Won't Fix", [18:18] "Confirmed", [18:18] "Triaged", [18:18] "In Progress", [18:18] "Fix Committed", [18:18] 'Fix Released']) [18:18] zoiks! [18:18] (I thought I had my newlines done differently) [18:19] I had to specify each status manually, because 'searchTasks' defaults to only finding open bugs [18:19] Then there's some code to calculate a percentage and print it out nicely [18:19] percentage = 100.0 * length(fixed_bugtasks) / length(total_bugtasks) [18:19] print "%s is %.2f%% successful on bugs in %s" % ( [18:19] reporter.display_name, percentage, pillar.display_name) [18:19] Let's try running it! [18:19] $ python bugstats.py ubuntu jml [18:19] Jonathan Lange is 22.22% successful on bugs in Ubuntu [18:19] $ python bugstats.py launchpad-code jml [18:19] Jonathan Lange is 56.16% successful on bugs in Launchpad Bazaar Integration [18:20] Again, you can try it with yourself, or with a friend, or with cjwatson [18:20] That's all I have for the example. [18:20] = 3. Gotchas = [18:20] There are quite a few things that can trip you up with the Launchpad API [18:20] We've already bumped into a couple of them: it's got bugs. [18:20] https://bugs.edge.launchpad.net/launchpadlib/ for more info on that [18:21] The second is that even though it does have documentation, it's not always easy to apply to what you need. [18:21] (wtf, "Colin Watson is 79.65% successful on bugs in Ubuntu"!) [18:21] I'm going to pimp http://help.launchpad.net/API again, since there really is a fair chunk of good documentation there. [18:21] https://help.launchpad.net/API/Examples is good, as is https://help.launchpad.net/API/launchpadlib [18:21] However, the docs often aren't up to scratch. [18:22] There is only one way for them to get better [18:22] People like your good selves must ask questions, get answers, and then edit the help.launchpad.net wiki. [18:22] Another gotcha is error messages. [18:22] Because launchpadlib is a very, very thin wrapper over a generic RESTful client library, if ever you get an exception raised, it's going to be really unhelpful. [18:22] https://help.launchpad.net/API/Examples#Get%20a%20useful%20error%20message%20from%20launchpadlib shows the work-around. [18:22] I think I filed a bug about that. [18:23] hmm. [18:23] there's another gotcha, which is *performance* [18:23] For a lot of the things you probably want to do, you'll end up writing code that looks like this: [18:23] for thing in bunch_of_things: [18:23] thing.do_something_on_launchpad() [18:23] Code like this is really really slow [18:23] It'll do an HTTPS request for each 'thing' [18:23] We'd like to have some way of reducing that load, but we don't have any good answers yet [18:23] Only two more gotchas left :) [18:23] Not everything that's in Launchpad itself is exposed via the API. [18:23] Sorry. [18:23] It's generally either really, really easy to expose stuff over the API or almost impossible. [18:24] If the thing you want falls into the first category, you can probably patch Launchpad yourself. [18:24] In any case, *file a bug*. Everything that's not exposed over the API is a bug. [18:24] Last gotcha: testing launchpadlib apps is hard [18:24] jkakar has done some work on this recently [18:24] but I haven't had a chance to look at it. [18:24] That's it on the gotchas: bugs, docs, errors, potato programming, unexposed methods, testing [18:25] = 4. Future = [18:25] There are plenty of other examples of launchpadlib apps out there [18:25] bughugger uses launchpadlib (https://launchpad.net/bughugger) [18:25] quickly has some templates for using launchpadlib (https://launchpad.net/quickly) [18:25] The code that puts branches in https://code.launchpad.net/ubuntu is done entirely with launchpadlib (we talked about those branches last session!) [18:25] There's also a stack of projects that are part of a group called 'lpx' that you can find at https://launchpad.net/lpx [18:25] and at http://help.launchpad.net/API/Uses there's even more stuff [18:25] If ever you need help, ask on #launchpad or #launchpad-dev on freenode or send an email to launchpad-users@lists.launchpad.net [18:26] That's it from me. [18:26] Questions? [18:26] Cool ideas for launchpadlib programs? [18:26] General complaints about Launchpad that I can respond soothingly to? [18:27] By "complaint", I of course meant "suggestion for improvement" [18:27] QUESTION: so, Python plays a HUGE part in Ubuntu, no? Other scripting won't feel left out? [18:27] A lot of Ubuntu stuff is written in Python. [18:28] As I mentioned earlier, you don't *have* to use Python to control Launchpad, but it makes it a lot easier. [18:28] QUESTION: Can you make more examples for python-snippets happen? [18:29] jonobacon, I can point you at https://help.launchpad.net/API/Examples again, I guess :) [18:29] QUESTION: Is there a way to access two JSON values in one "call"... I.e. launchpad.people[reporter_name].latitude gets the latitude -- is there a way to also get the longitude at the same time? [18:30] kamalmostafa, I'm fairly sure that if you get the object first, then subsequent attribute access doesn't do a separate webapp call. [18:30] so... [18:30] reporter = launchpad.people[reporter_name] [18:30] reporter.latitude, reporter.longtitude [18:30] QUESTION: is there a need to worry about python 3.0 breaking something? [18:30] vocx, wrt launchpadlib or wrt Ubuntu more generally? [18:31] vocx, launchpadlib is a Python 2 application. It hasn't been tested with Python 3. [18:32] vocx, I'm not the launchpadlib maintainer, but I tend to think of python3 as a completely different language. if a program works on both 2 and 3 it's a happy accident [18:33] vocx, no plans to port to 3 yet. [18:33] vocx, I guess we'd start doing that when Ubuntu stops providing Python 2 [18:34] are there any other questions? [18:34] QUESTION: Can we convince jonobacon to use metal umlauts in his name so my friends can go back to calling me Jono? === jml is now known as joneaux === joneaux is now known as jml [18:37] Last chance for questions folks [18:37] remember you can get the code for the example by 'bzr branch lp:~jml/+junk/bugstats' [18:37] and that the folk on #launchpad-dev are helpful and friendly and want more people doing stuff with launchpadlib [18:38] ok folks }}}