Ubuntu Wiki

Dev Week -- Stop Screen-Scraping, Use the Launchpad Web Service API -- leonardr -- Tue, Jan 20

UTC -4(EST)

(01:02:13 PM) leonardr: ok, i'll get started now
(01:02:29 PM) leonardr: Hi all.
(01:02:35 PM) leonardr: My name is Leonard Richardson. I'm on the Launchpad Foundations team and I'm the co-author of the O'Reilly book "RESTful Web Services".
(01:02:55 PM) leonardr: I'm here to talk about the Launchpad web service API--how to use it and what advances have been made since the last UDW.
(01:03:02 PM) leonardr: I'll do an infodump and then take your questions.
(01:03:07 PM) leonardr: If you have questions in the meantime just put them in #ubuntu-classroom-chat.
(01:03:14 PM) leonardr: Most of this infodump will be familiar to those of you who were at this chat at the last UDW. I ask you to bear with me so I can get everyone up to speed.
(01:03:22 PM) leonardr:  
(01:03:26 PM) leonardr: 1. Intro
(01:03:35 PM) leonardr: First thing to know is that we've got docs talking about the API here: https://help.launchpad.net/API
(01:03:43 PM) leonardr: Put simply, we've created an easy way for you to integrate Launchpad into your own applications.
(01:03:51 PM) leonardr: If you perform the same tasks on Launchpad over and over again, you can now write a script that automates the tasks for you.
(01:04:04 PM) leonardr: You don't have to rely on fragile screen-scraping.
(01:04:09 PM) leonardr: If you're a developer of an IDE, testing framework, or some other program that has something to do with software development, you can integrate Launchpad into the program to streamline the development processes.
(01:04:16 PM) leonardr: If you run a website for a project hosted on Launchpad, you can get project data from Launchpad and publish it on your website.
(01:04:24 PM) leonardr: And so on. You'll eventually be able to use the API to do most of the things you can do through the Launchpad web site.
(01:04:28 PM) leonardr:  
(01:04:32 PM) leonardr: 2. Tools
(01:04:38 PM) leonardr: The simplest way to integrate is to use launchpadlib, a Python library we've written.
(01:04:44 PM) leonardr: (see https://help.launchpad.net/API/launchpadlib)
(01:04:49 PM) leonardr: This gives you a Pythonic interface to the Launchpad API, so you don't have to know anything about HTTP client programming:
(01:04:56 PM) leonardr: >>> launchpad.me.name
(01:04:56 PM) leonardr: u'leonardr'
(01:04:56 PM) leonardr: >>> launchpad.bugs[1].title
(01:04:56 PM) leonardr: u'Microsoft has a majority market share'
(01:05:04 PM) leonardr: But it's also easy to learn the API's HTTP-based protocol and write your own client in some other language.
(01:05:08 PM) leonardr: (see https://help.launchpad.net/API/Hacking)
(01:05:19 PM) leonardr:  
(01:05:23 PM) leonardr: 3. Progress and Roadmap
(01:05:29 PM) leonardr: At the last UDW I said that the web service publishes information about people, bugs, and the Launchpad registry, (the projects, milestones, etc.).
(01:05:37 PM) leonardr: Since then, the bugs work has been more or less completed. CVEs have also been published through the web service.
(01:05:50 PM) leonardr: Code branches have come online.
(01:05:55 PM) leonardr: Merge proposals have been published but are currently read-only. They should be read-write within a week.
(01:06:04 PM) leonardr: Additional bits of the registry, like milestones, have been published since the last UDW.
(01:06:10 PM) leonardr: Some of the Soyuz functionality has come online: archives and archive permissions.
(01:06:16 PM) leonardr: The hardware database will be the next thing to be published, hopefully within a month.
(01:06:25 PM) leonardr: Publication through the web service is still not a priority for translations, answers, and blueprints.
(01:06:44 PM) leonardr: so, that's the infodump, i invite your questions
(01:07:38 PM) leonardr: <sianis> QUESTION: do you plan write API for Rosetta?
(01:07:41 PM) ara left the room (quit: "Ex-Chat").
(01:08:09 PM) leonardr: so, the team that does some part of launchpad, like rosetta, is also responsible for publishing that part of launchpad through the web service
(01:08:41 PM) leonardr: i talked to the translations team today and they don't have plans to publish translations through the web service, per se
(01:08:58 PM) leonardr: however, we will also be using the web service to add ajax functionality to the launchpad website
(01:09:22 PM) leonardr: and they will be publishing some of the translations functionality for that, so you will get at least some of the translations
(01:09:42 PM) leonardr: but there's no plan right now to publish the whole things as there was for bugs--what they publish might be useful to you, it might not
(01:10:04 PM) leonardr: <bullgard4> QUESTION: What is a " Pythonic interface "?
(01:10:53 PM) leonardr: a pythonic interface is one that "looks like python". it presents objects that act like built-in python data structures, as opposed to (eg.) one with a lot of getters and setters
(01:11:04 PM) leonardr: <thekorn> Question: can you please explain (in short) the different access-level?
(01:11:04 PM) leonardr: <thekorn> and especially: why is there a "no access level"
(01:11:18 PM) leonardr: is there a no access level? let me check
(01:11:48 PM) leonardr: i'm opening up the list of access levels
(01:12:01 PM) leonardr: all right
(01:12:21 PM) leonardr: so, we've set up authorization to the launchpad web service using the oauth standard
(01:12:40 PM) leonardr: which is basically a way for the end-user to delegate a certain amount of authority to a program
(01:13:09 PM) leonardr: the problem is that although you trust your web browser enough to type in your launchpad password
(01:13:45 PM) leonardr: you probably don't extend that same level of trust to other programs, like IDEs or "portal" websites that promise to show you a dashboard of your launchpad activity
(01:13:57 PM) leonardr: you shouldn't trust those sites with your launchpad password
(01:14:09 PM) leonardr: oauth is a way to delegate a certain amount of power to those sites, without giving them your password
(01:14:26 PM) leonardr: in that context, the access levels are the different amounts you can trust a particular site or application
(01:14:50 PM) leonardr: if a site promises to make a dashboard for you, then you can give the site read-only access to your data
(01:14:56 PM) leonardr: it has no business modifying your data
(01:15:11 PM) leonardr: similarly, you can authorize an application to only access public data on your behalf
(01:15:42 PM) leonardr: as for the 'no access' level, flacoste said in classroom-chat:
(01:15:47 PM) leonardr: <flacoste> thekorn: say a rogue application try to ask for a token, the user can click on no access level to dismiss the request
(01:15:51 PM) leonardr: it's a 'cancel' button
(01:16:03 PM) leonardr: hopefully that provides at least some enlightenment
(01:16:20 PM) leonardr: <maxb> QUESTION: Will it become possible to use launchpadlib without needing to login, if all you want is read-only programmatic access to public data?
(01:16:52 PM) leonardr: our initial decision was to prohibit this altogether. we're now reconsidering but we haven't made a decision one way or the other yet
(01:17:05 PM) leonardr: <ia> QUESTION: could you just name, please, a few apps, which uses LP API?
(01:17:18 PM) leonardr: i think thekorn could name more than i can
(01:17:41 PM) leonardr: i'm also bad with names
(01:17:46 PM) thekorn: there are some tools switching from screenscraping to lplib,
(01:17:50 PM) leonardr: <flacoste> the is a bzr plugin for eclipse
(01:17:57 PM) thekorn: like ubuntu-dev-tools
(01:19:05 PM) leonardr: also, for every publicly available tool there are several private scripts
(01:19:26 PM) leonardr: people write everyday scripts to get around whatever annoys them about launchpad or to do batch operations
(01:19:49 PM) leonardr: <maxb> QUESTION: Rationale for prohibiting launchpadlib access to data which can be screen-scraped without auth would be interesting?
(01:20:53 PM) leonardr: there's a general worry about access to the web service getting out of control
(01:21:24 PM) leonardr: we have various ways to throttle usage for different scenarios, and one of the scenarios is when one person is using too many resources, whether by accident, due to a bug, or maliciously
(01:21:41 PM) leonardr: in that case we'd throttle all access by that person
(01:21:51 PM) leonardr: like i said, we're reconsidering it
(01:22:18 PM) leonardr: <istaz_> QUESTION: Is there a way to ask for change to be Pushed to you  instead of having to poll?
(01:22:32 PM) leonardr: no, not really
(01:22:37 PM) leonardr: are you thinking something like email?
(01:23:01 PM) leonardr: in general the web works on polling, even if you're just polling a summary of what's changed
(01:23:12 PM) leonardr: <istaz_> leonardr: yes
(01:23:24 PM) leonardr: no, there's no email or other kind of push architecture
(01:24:08 PM) leonardr: if you're worried about bandwidth usage, there are standard http techniques like conditional get that we use and plan to use more of
(01:24:25 PM) leonardr: launchpadlib automatically uses them
(01:25:33 PM) leonardr: xmpp is an interesting possibility but i don't think you should expect it anytime soon
(01:27:55 PM) leonardr: any other questions?
(01:28:55 PM) leonardr: <thekorn> Question: is there any work on a javascript implementation of the API, like launchpadlib for python
(01:28:58 PM) leonardr: yes, there is, actually
(01:29:24 PM) leonardr: because as i said, we are going to be using the web service to add ajax ui elements to launchpad
(01:29:40 PM) leonardr: you can actually see it now
(01:30:11 PM) leonardr: https://launchpad.net/+icing/rev7479/launchpad-ajax.js
(01:30:20 PM) leonardr: but right now it only works on pages on launchpad itself
(01:30:45 PM) leonardr: this is partly because of the design and partly because of the 'same-origin policy' that makes it difficult to have a page on foo.com that makes an XMLHTTPRequest to bar.com
(01:31:17 PM) leonardr: it's not a huge priority for us but eventually the javascript library should be usable outside of launchpad
(01:31:27 PM) leonardr: however, note that it's very different from launchpadlib
(01:31:54 PM) leonardr: because it doesn't have access to the wadl file, you have to know a lot more about the url structure and what the web service looks like on the http level
(01:33:21 PM) leonardr: anything else?
(01:36:04 PM) leonardr: ok, i'll stick around until the next presentation in case there are any more questions
(01:36:50 PM) leonardr: <creek23> QUESTION: Can we extract XML for Flash display?
(01:37:30 PM) leonardr: the web service serves JSON, not xml
(01:37:48 PM) leonardr: it's also pretty likely that a flash application will run afoul of the same-domain policy, just like a javascript application
(01:38:18 PM) leonardr: if you can get around the same-domain policy somehow, there's got to be a json parser written in flash
(01:38:41 PM) leonardr: but that's not a case we're really thinking about