Ubuntu Wiki

Dev Week -- Using the LP API for fun and profit -- leonardr -- Tue Sep 1st, 2009

UTC

(03:02:29 PM) didrocks: now, the stage is opened to leonardr
(03:02:35 PM) leonardr: thanks didrocks
(03:02:41 PM) didrocks: he will teach you about blackmagic on pythonlaunchpadlib :)
(03:03:10 PM) leonardr: My name is Leonard Richardson. I'm on the Launchpad Foundations team and I'm the co-author of the O'Reilly book "RESTful Web Services".
(03:03:16 PM) leonardr: I'm here to talk about the Launchpad web service API--how to use it and what advances have been made since the last UDW.
(03:03:31 PM) leonardr: I'll do an infodump for a few minutes and then take your questions for the rest of the hour.
(03:03:43 PM) leonardr: If you have questions during the infodump, just put them in #ubuntu-classroom-chat.
(03:03:51 PM) leonardr: I give this infodump at every UDW, so it may be familiar to you. I ask you to bear with me so I can get everyone up to speed.
(03:04:02 PM) leonardr: 1. Intro
(03:04:15 PM) leonardr: First thing to know is that we've got docs talking about the API here: https://help.launchpad.net/API
(03:04:19 PM) leonardr: Put simply, we've created an easy way for you to integrate Launchpad into your own applications.
(03:04:30 PM) leonardr: f you perform the same tasks on Launchpad over and over again, you can write a script that automates the tasks for you.
(03:04:38 PM) leonardr: You don't have to rely on fragile screen-scraping.
(03:04:43 PM) leonardr: If you're a developer of an IDE, testing framework, or some other program that has something to do with software development, you can integrate Launchpad into the program to streamline the development processes.
(03:04:49 PM) leonardr: If you run a website for a project hosted on Launchpad, you can get project data from Launchpad and publish it on your website.
(03:04:55 PM) leonardr: And so on. You can use the API to do most of the things you can do through the Launchpad web site.
(03:05:15 PM) leonardr: 2. Tools
(03:05:29 PM) leonardr: The simplest way to integrate is to use launchpadlib, a Python library we've written.
(03:05:34 PM) leonardr: see https://help.launchpad.net/API/launchpadlib, ubuntu package python-launchpadlib)
(03:05:45 PM) leonardr: This gives you a Python-idiomatic interface to the Launchpad API, so you don't have to know anything about HTTP client programming:
(03:05:56 PM) leonardr: >>> launchpad.me.name
(03:05:57 PM) leonardr: u'leonardr'
(03:05:57 PM) leonardr: >>> launchpad.bugs[1].title
(03:05:57 PM) leonardr: u'Microsoft has a majority market share'
(03:06:05 PM) leonardr: But it's also easy to learn the API's HTTP-based protocol and write your own client in some other language.
(03:06:09 PM) leonardr: (see https://help.launchpad.net/API/Hacking)
(03:06:27 PM) leonardr: 3. Progress and Roadmap
(03:06:32 PM) leonardr: At the last UDW I said that the web service publishes information about people, bugs, code branches, archives, and the launchpad registry (the projects, milestones, etc.)
(03:06:38 PM) leonardr: Here's what's been published since then (AFAIK):
(03:06:44 PM) leonardr: * The translation import queue (more or less read-only)
(03:06:51 PM) leonardr: * Lots of distro stuff i don't understand, like package uploads and package sets
(03:06:59 PM) leonardr: * Merge proposals and code reviews
(03:07:05 PM) leonardr: * Project releases
(03:07:09 PM) leonardr: * The hardware database
(03:07:19 PM) leonardr: The future:
(03:07:27 PM) leonardr: Publication through the web service is still not a priority for translations (apart from the import queue, which is done), answers, or blueprints.
(03:07:38 PM) leonardr: Work on publishing new things on the LP web service has slowed down since the last UDW, since most of Launchpad is published now.
(03:07:47 PM) leonardr: Right now I'm working on making the underlying library (lazr.restful) an attractive option for anyone who wants to publish a web service.
(03:07:53 PM) leonardr: We are also working on improving the client.
(03:08:03 PM) leonardr: So, that's the infodump, i invite your questions.
(03:08:20 PM) leonardr: <rugby471> leonardr: I have used python-launchpadlib in memaker and I loved it's simplicity, however the one gripe I had with it was the very first authentication request the user has to do with launchpad, is there any plans/thoughts on how to make this a bit less clunky?
(03:08:59 PM) leonardr: rugby471: yes
(03:09:16 PM) leonardr: this work is tracked in bug 387297
(03:10:09 PM) leonardr: basically, the workflow is the way it is because we wanted to exploit the fact that the user already trusts their web browser with their launchpad credentials, where (no offense) they don't trust your application
(03:10:56 PM) leonardr: because developers dislike this workflow to the extent that they're hacking around it, we've decided to create some alternate trusted clients
(03:11:24 PM) leonardr: instead of opening up the user's web browser, you'll be able to run a console trusted client app, a gtk client app, etc
(03:11:33 PM) leonardr: it'll be similar to the pinentry application
(03:11:48 PM) leonardr: this is kind of on the back burner while i work on lazr.restful, but it is in progress
(03:12:06 PM) leonardr: any other questions?
(03:13:05 PM) leonardr: <frandieguez__> QUESTION: are there libraries for other languages, like ruby?
(03:13:25 PM) leonardr: the only other library i know of is the one we wrote in javascript to use in launchpad ajax code
(03:14:07 PM) leonardr: if you know of a good wadl library for a language, it's not difficult to write a launchpadlib-like library on top of it
(03:14:16 PM) leonardr: just read in the wadl file that describes launchpad's capabilities
(03:14:22 PM) leonardr: however, wadl libraries are not very common
(03:15:14 PM) leonardr: in addition to taking questions, if anyone attending this chat has written a library that uses launchpadlib, or integrated the web service into an applicatino, i'd like to hear about it
(03:15:27 PM) leonardr: i'll collate what you say and put it in at the end so people can see what's been done already
(03:17:48 PM) leonardr: #ubuntu-classroom-chat is a little quiet, but i'll be here until bdmurray takes over in 40 minutes
(03:19:22 PM) leonardr: as long as we're here i'll talk a bit about the server side of things
(03:19:26 PM) leonardr: which is what i'm working on now
(03:20:00 PM) leonardr: i'm running a project called lazr.restful
(03:20:03 PM) leonardr: https://edge.launchpad.net/lazr.restful
(03:20:16 PM) leonardr: this used to be part of launchpad and was split out and made open source a few months before launchpad was
(03:20:40 PM) leonardr: it's a zope application that publishes data model objects through a web service
(03:20:54 PM) leonardr: for the past couple of months i've been working on making it appealing to people who don't use zope
(03:21:33 PM) leonardr: right now we have the zope application hidden behind a piece of wsgi middleware
(03:21:47 PM) leonardr: and i've been able to use it internally to publish django model objects
(03:22:03 PM) leonardr: so pretty soon it should be usable by the zope-phobic
(03:22:24 PM) leonardr: <tormod> QUESTION: are there some "hello world" examples on say, how to mass-process bugs etc
(03:22:54 PM) leonardr: there are several example scripts in launchpadlib/examples, and there's a big pile of them about to be added to that directory (i'm not sure from where)
(03:23:01 PM) leonardr: let me see if there's one on there dealing with bugs
(03:23:59 PM) leonardr: it's actually samples/, not examples/, and there is no bugs code there
(03:24:27 PM) leonardr: i can work up something live, right now
(03:25:15 PM) leonardr: so, i've got a python session going, and i've got the api reference in my web browser
(03:25:35 PM) leonardr: that's https://edge.launchpad.net/+apidoc/
(03:25:55 PM) leonardr: two steps to operate on a bunch of bugs: 1. identify the bugs, 2. operate on them
(03:27:07 PM) leonardr: <tormod> if you need an example, change all xorg.0.logs to plain/text :)
(03:27:10 PM) leonardr: ok, i'll do what i can
(03:27:16 PM) leonardr: we start by navigating to the bugs
(03:27:23 PM) leonardr: >>> xorg = launchpad.projects['xorg']
(03:28:24 PM) leonardr: i'm looking at https://edge.launchpad.net/+apidoc/#project for something that will help
(03:28:34 PM) leonardr: (i don't actually know the lp api very well, as i only wrote the framework)
(03:28:47 PM) leonardr: i think it might be smarter to go in through https://edge.launchpad.net/+apidoc/#bugs
(03:30:44 PM) leonardr: ok, an introspection method, xorg.lp_operations, shows an operation called 'searchTasks'
(03:30:50 PM) leonardr: that'll get us closer to the bugs
(03:31:43 PM) leonardr: i'll just get the new tasks:
(03:31:45 PM) leonardr: >>> tasks = xorg.searchTasks(status="New")
(03:32:50 PM) leonardr: now we iterate over the tasks, find any attachments, and hack the attachments as per tormod's example
(03:34:28 PM) leonardr: so let's take one task as an example; then the rest is just iteration
(03:35:03 PM) leonardr: we have the task. the bug attachments are on the bug
(03:35:20 PM) leonardr: 'bug' is a property of bug_task, so: bug = task.bug
(03:35:53 PM) leonardr: each bug has a collection of attachments, so we iterate over bug.attachments
(03:36:14 PM) leonardr: i'm just picking the first task, the first attachment, etc
(03:36:18 PM) leonardr: so the code i've run so far is:
(03:36:33 PM) leonardr: bugs = [x.bug for x in tasks]
(03:36:37 PM) leonardr: bug = bugs[0]
(03:36:53 PM) leonardr: attachment = [x for x in bug.attachments][0]
(03:37:17 PM) leonardr: tormod points out:
(03:37:18 PM) leonardr: <tormod> I can not find "mime type" under https://edge.launchpad.net/+apidoc/#bug_attachment
(03:37:48 PM) leonardr: that's because the bug_attachment object isn't the actual attachment
(03:38:00 PM) leonardr: it's an entry in the database containing information about the attachment
(03:38:07 PM) leonardr: the actual attachment is available as attachment.data
(03:38:31 PM) leonardr: if you get attachment.data in launchpadlib you'll have a HostedFile object
(03:39:05 PM) leonardr: i need to look up how these work...
(03:39:10 PM) leonardr: i'm looking in https://help.launchpad.net/API/launchpadlib
(03:39:23 PM) leonardr: https://help.launchpad.net/API/launchpadlib#Hosted%20files
(03:39:42 PM) leonardr: rugby471 is familiar with these objects since a mugshot works the same way
(03:40:03 PM) leonardr: we can get a filehandle on the object by opening it for read
(03:40:09 PM) leonardr: and the mime type is an attribute on the filehandle
(03:40:47 PM) leonardr: >>> handle = attachment.data.open()
(03:40:57 PM) leonardr: >>> handle
(03:40:57 PM) leonardr: <launchpadlib.resource.HostedFileBuffer instance at 0x9e2a72c>
(03:41:03 PM) leonardr: >>> handle.filename
(03:41:03 PM) leonardr: 'Xorg.0.log'
(03:41:07 PM) leonardr: >>> handle.content_type
(03:41:07 PM) leonardr: 'text/plain'
(03:41:44 PM) leonardr: so that content type is ok
(03:41:53 PM) leonardr: but let's say we wanted to change it
(03:42:01 PM) leonardr: because these files are stored in the launchpad librarian, you can't just change the content type and save, the way you can change a bug's description
(03:42:07 PM) leonardr: you need to create a new librarian file
(03:42:36 PM) leonardr: basically, open the filehandle for write, specifying the correct content type and filename. then write the content to the filehandle and close it
(03:42:47 PM) leonardr: something like this
(03:43:05 PM) leonardr: >>> old_name = handle.filename
(03:43:11 PM) leonardr: >>> old_content = handle.read()
(03:43:33 PM) leonardr: >>> write_handle = attachment.data.open("w", "text/plain", old_name)
(03:43:37 PM) leonardr: >>> write.handle.write(old_content)
(03:43:41 PM) leonardr: >>> write_handle.close()
(03:43:59 PM) leonardr: not the most convenient code (because librarian files are write-once), but it's possible
(03:44:29 PM) leonardr: so you'd do that for every bug in the xorg project that met your criteria
(03:45:04 PM) leonardr: unfortunately there's no way to directly specify your criteria "bugs that have an attachment called Xorg.0.log with a content type other than text/plain"
(03:45:31 PM) leonardr: you'll need to use some cruder criteria, such as new bugs, as i used
(03:48:19 PM) leonardr: ^arky^ requested the code i just put up on pastebin: http://pastebin.ubuntu.com/263382/
(03:48:35 PM) leonardr: no guarantees that will work as is, but by using dir() you should be able to make it work
(03:48:48 PM) leonardr: and that's how i proceed in general when writing these scripts
(03:49:07 PM) leonardr: i don't know much about lp, but i use the introspection methods and the api doc to zoom in on where i want to be
(03:49:22 PM) leonardr: <tormod> do you have to delete the old file from the librarian then? or are they indexed by filename?
(03:49:59 PM) leonardr: you can delete attachment.data with attachment.data.delete(). i don't think it's necessary. the old attachment will just stop beign referenced and will eventualyl be garbage collected
(03:50:50 PM) leonardr: i believe they're indexed by a unique id that you never see. that's how there can be 10,000 different attachments all called Xorg.0.log
(03:55:37 PM) leonardr: hm, i've been talking in classroom-chat by mistake
(03:55:41 PM) leonardr: i'll paste in a few q & as
(03:55:55 PM) leonardr: <^arky^> QUESTION: Can we use python-launchpadlib (1.5.1-0ubuntu1) to try these code examples
(03:56:01 PM) leonardr: ^arky^: yes, but the first time you run one of these scripts you will need to give launchpadlib permission to access your launchpad account
(03:56:08 PM) leonardr: <tormod> what was dir() about?
(03:56:14 PM) leonardr: tormod: you can use dir() on any launchpadlib object to see its capabilities are
(03:56:24 PM) leonardr: so we also created a bunch of special introspection properties
(03:56:29 PM) leonardr: described here: https://help.launchpad.net/API/launchpadlib#Getting%20help
(03:56:36 PM) leonardr: a quick example of the introspection properties:
(03:56:36 PM) leonardr: >>> launchpad.people['leonardr'].lp_entries
(03:56:36 PM) leonardr: ['preferred_email_address', 'archive', 'team_owner', 'mugshot']
(03:56:36 PM) leonardr: lp_entries shows you the individual objects associated with some other object
(03:56:36 PM) leonardr: in this case, an email address, an archive, another person (if leonardr were a team, it might have an owner), and a binary file (mugshot)
(03:57:10 PM) leonardr: it's almost time for bdmurray to take over, so any last minute questions
(03:59:50 PM) leonardr: before i go, one example of an app that integrates into launchpad, from rugby471
(03:59:53 PM) leonardr: I have integrated it into memaker for one click updating of the user's mugshot ( package memaker in karmic)
(04:00:23 PM) leonardr: one more question before he takes over
(04:00:25 PM) leonardr: <thekorn> leonardr, QUESTION: is the ability to change the content of an attachment a feature or a bug?
(04:00:47 PM) leonardr: i don't know. you'd need to ask the bugs team. my guess is it's a feature
(04:01:24 PM) leonardr: in general, it's a feature that you can replace the contents of a hosted file (like a mugshot), but maybe the bugs team wants you to upload a new attachment rather than modifying one
(04:03:03 PM) leonardr: bdmurray, i yield the floor