Ubuntu Wiki

Ubuntu Open Week - Upstreaming Bugs - Jorge Castro - Wed, Nov 5th, 2008

(01:58:50 PM) jcastro: ok I will start in about ~2 minutes
(02:01:02 PM) jcastro: ok, thanks everyone for coming, my name is Jorge Castro and for the next hour I will be talking about upstreaming bugs
(02:01:24 PM) jcastro: Hopefully if you're at this session you've attended bdmurray and pedro's talks about bugs
(02:01:39 PM) jcastro: but if not hopefully you can get something out of this session anyway. :D
(02:01:47 PM) jcastro: So what exactly do I mean by upstreaming bugs?
(02:02:07 PM) jcastro: First off, what we ship as "Ubuntu" contains a bunch of software written by other open source projects
(02:02:20 PM) jcastro: this can be large projects like linux, GNOME, KDE, X.org, and others
(02:02:35 PM) jcastro: but also includes smaller projects that you might not have heard of before
(02:03:03 PM) jcastro: Ubuntu fits into the picture by bundling this software and shipping it as a distribution every 6 months
(02:03:37 PM) jcastro: As such it is up to us to make sure that when our users report bugs that the bug report makes its way to the right developer so that they can address it.
(02:04:00 PM) jcastro: We do this in our bug tracker, launchpad, which has the ability to link Ubuntu bugs with upstream bug trackers.
(02:04:32 PM) jcastro: So in this session I will go over some workflows on how to do this efficiently, and give you some examples of ideal bugs that we've forwarded.
(02:04:35 PM) jcastro: Any questions so far?
(02:04:52 PM) jcastro: (Please ask in #ubuntu-classroom-chat)
(02:05:08 PM) jcastro: Ok, moving on ...
(02:05:46 PM) jcastro: For what I call the "top100 projects" in ubuntu, we generate a chart that shows us how well we're linking bugs
(02:05:58 PM) jcastro: in this instance it's the "top100 projects sorted by open bugs"
(02:06:14 PM) jcastro: we make this chart available here: https://launchpad.net/ubuntu/+upstreamreport
(02:06:19 PM) jcastro: We call this the "upstream report"
(02:06:52 PM) jcastro: what this report allows us to do is to glance at a large cross section of the distro to see how well we're doing overall, and more importantly, to find out where we need to improve
(02:07:11 PM) jcastro: So everyone if you would, click that link so you can follow along
(02:07:26 PM) jcastro: So let's look at an example
(02:07:35 PM) jcastro: I will just pick openoffice.org, which is the third row down
(02:07:54 PM) jcastro: As you can see, right now there are 523 open bugs in openoffice.org in Ubuntu
(02:08:15 PM) jcastro: that means that right now there are 523 possible defects that need to be addressed.
(02:08:38 PM) jcastro: Of these, 364 are in a triaged state, which means that a developer has looked at it and has determined that it is indeed a bug.
(02:08:55 PM) jcastro: The upstream column is where it gets interesting
(02:09:02 PM) jcastro: That number is 338.
(02:09:25 PM) jcastro: That means that out of those 523 bugs reported, a developer has determined that 338 of them are bugs in OpenOffice itself
(02:10:16 PM) jcastro: of those 338, 315 have a bug link in launchpad to the openoffice bug tracker
(02:10:33 PM) jcastro: this means that we have a 93.20% linkage rate
(02:10:41 PM) jcastro: which is very good (we always want to be over 90%)
(02:11:15 PM) jcastro: so basically, when a developer determines that a bug is upstream, they open an upstream task in ubuntu, which then shows up in the upstream column
(02:11:33 PM) jcastro: then it's up to bug people like us to make sure that these bugs get linked upstream, which I will show you in a minute
(02:12:06 PM) jcastro: So, what really you need to be concerned about is the last column
(02:12:19 PM) jcastro: these are bugs that a developer has deemed to be "upstreamable"
(02:12:28 PM) jcastro: but has NOT been linked to an upstream bug report
(02:12:45 PM) jcastro: as an example, if you scroll down to "kdebase"
(02:12:55 PM) jcastro: you'll see that the last column is a 12.
(02:12:57 PM) jcastro: click on the 12
(02:13:09 PM) jcastro: and it takes you to this list: https://bugs.edge.launchpad.net/ubuntu/+source/kdebase/+bugs?search=Search&field.status_upstream=pending_bugwatch
(02:13:37 PM) jcastro: So these 12 bugs are something that kubuntu developers and QA people have determined to be an upstream issue
(02:13:57 PM) jcastro: so let's take a look at one
(02:14:04 PM) jcastro: please click on bug 172682
(02:14:14 PM) jcastro: which takes you here: https://bugs.edge.launchpad.net/ubuntu/+source/kdebase/+bug/172682
(02:14:15 PM) ubot5`: Launchpad bug 172682 in kdebase "Dolphin cannot open a tar file on a smb share" [Wishlist,Confirmed]
(02:14:54 PM) jcastro: So, as you read the bug report you will find where Jonathan says: "The best bet for getting this feature is to file a wishlist report at http://bugs.kde.org. If you (or anybody) files a wishlist it would also be handy to link the report here or at least post the link to the wishlist in the comments."
(02:15:10 PM) jcastro: So ideally, you would go to the KDE  bugtracker
(02:15:27 PM) jcastro: and then first search to see if a bug is already filed.
(02:16:00 PM) jcastro: If it is, you would get the url to that bug, click "Also affects project" and paste the bug url in there
(02:16:11 PM) jcastro: at that point launchpad will make a bug watch to monitor the bug
(02:16:24 PM) jcastro: if in the future the bug is fixed the bug watch will reflect that
(02:16:39 PM) jcastro: And at that point the bug is considered fixed upstream, but NOT yet in ubuntu
(02:16:54 PM) jcastro: at that point the bug would show up in the harvest tool so a developer can look at it
(02:17:08 PM) jcastro: I have an example of this just from today
(02:17:19 PM) jcastro: https://bugs.launchpad.net/ubuntu/+source/brasero/+bug/292201
(02:17:21 PM) ubot5`: Launchpad bug 292201 in brasero "brasero don't add files to data project by drag and drop" [Low,Fix committed]
(02:17:45 PM) jcastro: So this bug was reported on november 1st.
(02:18:09 PM) jcastro: at one point Pedro decided that this was a bug in brasero itself, not in anything that we've done in Ubuntu
(02:18:17 PM) jcastro: so he filed a bug in the upstream GNOME bugzilla
(02:18:37 PM) jcastro: then he pasted the link in a comment, clicked "Also affects project" and pasted the GNOME bugzilla URL in there
(02:18:59 PM) jcastro: if you look at the top of the bug report you will see that it lists "gnome-bugs #559173"
(02:19:14 PM) jcastro: If you click on that you can go see the report in the GNOME bugzilla
(02:19:35 PM) jcastro: If you see Pedro's report here: http://bugzilla.gnome.org/show_bug.cgi?id=559173
(02:19:37 PM) ubot5`: Gnome bug 559173 in general "brasero don't add files to data project by drag and drop" [Normal,Resolved: fixed]
(02:19:48 PM) jcastro: you will see that he also put a link to the launchpad bug in that report
(02:20:05 PM) jcastro: so if you check both pages you'll see this workflow
(02:20:18 PM) jcastro: reporter -> pedro -> brasero developer
(02:20:39 PM) jcastro: Basically pedro is being a link between an ubuntu user and a GNOME developer
(02:20:56 PM) jcastro: the great thing is, anyone can be this link!
(02:21:06 PM) jcastro: so what I tell people is to pick a small project to start off with
(02:21:12 PM) jcastro: something you want to care about
(02:21:28 PM) jcastro: and then go through it's bugs in ubuntu, and search for those bugs in the upstream bug tracker
(02:21:37 PM) jcastro: and then make the link in launchpad and in the upstream bug tracker
(02:21:47 PM) jcastro: This helps get information to upstream developers faster
(02:22:02 PM) jcastro: and ultimately will help get fixes out to the user faster, which is what it's all about
(02:22:15 PM) jcastro: QUESTION: when clicking on Also affects project, the form asks for a project name, do we just paste the bug url anyway?
(02:22:33 PM) jcastro: Sometimes some information for a certain package is not filled out properly in launchpad
(02:22:41 PM) jcastro: It won't let you paste a url into that field
(02:23:04 PM) jcastro: what I do is fill in the project name, hit search, then click on the title, and then it will fix that project and allow you to put a URL in
(02:23:10 PM) rick_h_: QUESTION: will the status for the gnome-bugs link in launchpad automatically update?
(02:23:25 PM) jcastro: thanks rick_h_
(02:23:27 PM) jcastro: Yes
(02:23:40 PM) jcastro: in that brasero example it has not updated yet, which is why it's unknown
(02:23:54 PM) rick_h_: QUESTION: How do you find out if the bug really also affects upstream and is not ubuntu-related?
(02:23:58 PM) jcastro: We use a tool called harvest as a "radar" that shows us when something is fixed upstream
(02:24:14 PM) jcastro: That is a tough question, probably the crux of the matter
(02:24:32 PM) jcastro: If you use the last column in the report and look at those bugs
(02:24:47 PM) jcastro: those will already be determined to be upstream, so you're safe to look at those and make linkages
(02:24:54 PM) jcastro: with some bugs it's easy
(02:25:07 PM) jcastro: usually if it's a feature request or a wishlist it's very much an upstream bug
(02:25:23 PM) jcastro: so "I wish firefox did this and that" is probably an upstream bug
(02:25:32 PM) jcastro: "firefox crashing" would be a harder one to solve
(02:25:52 PM) jcastro: which is why you should consult people in #ubuntu-bugs or the corresponding team for that package
(02:26:12 PM) jcastro: still, there are plenty of bugs in the last column that you can link that are already determined to be upstream
(02:26:16 PM) jcastro: so there's no shortage there
(02:26:53 PM) jcastro: The way I do it is just work off the last columns, and let smarter people make the decision on wether it's upstream or not. :D
(02:28:01 PM) jcastro: < homy> sorry, I don't understand that "last column" jcastro talks about.
(02:28:04 PM) jcastro: ok no problem
(02:28:05 PM) rick_h_: jcastro: can you post the list to the page with the "last column" again?
(02:28:15 PM) jcastro: https://edge.launchpad.net/ubuntu/+upstreamreport
(02:28:23 PM) jcastro: that last column on the upstream report
(02:28:30 PM) jcastro: That is the list of bugs I look at
(02:28:46 PM) jcastro: Ok, so how does this all tie together?
(02:29:19 PM) jcastro: When a bug watch shows that a bug is fixed upstream it shows up in Harvest
(02:29:30 PM) jcastro: the harvest page for brasero looks like this: http://daniel.holba.ch/harvest/handler.py?pkg=brasero
(02:29:53 PM) jcastro: so basically, in the past 2 months there have been 6 watches that have been shown as fixed
(02:30:07 PM) jcastro: the reason we have this harvest page
(02:30:34 PM) jcastro: is that an ubuntu developer can check on each of those bugs, find the fix, and fix the package in ubuntu
(02:30:56 PM) jcastro: so basically, I see it this way
(02:31:12 PM) jcastro: if you make links to upstream trackers, as those get fixed, they start to show up on harvest
(02:31:25 PM) jcastro: we basically are what I call "feeding the harvest machine"
(02:31:44 PM) jcastro: from there ubuntu developers have a nice little list of possible fixes that they can look at to fix the packages in ubuntu
(02:32:04 PM) jcastro: so really, if you know how to search for bugs in launchpad and  bugzilla
(02:32:19 PM) jcastro: just making the link between the two can really help developers find the right fixes
(02:32:59 PM) jcastro: while I am at it, I did a blog post on how this all works here:
(02:33:17 PM) jcastro: http://stompbox.typepad.com/blog/2008/09/introducing-the.html
(02:33:24 PM) jcastro: that has screenshots and examples as well
(02:33:36 PM) jcastro: So if you're lost right now you can refer to that document and the logs from this session
(02:33:49 PM) jcastro: < homy> ah, I think I understood: when I click on the "40" in the firefox-3.0 column for example, I get all upstreamable firefox-3.0 bugs  that don't have an upstream bug report yet?
(02:33:53 PM) jcastro: yes! correct!
(02:34:04 PM) jcastro: Now
(02:34:18 PM) jcastro: the upstream report is a great tool for finding possible upstream targets
(02:34:27 PM) jcastro: but like all tools, you can use it to cause havok as well
(02:34:50 PM) jcastro: so just because you see 40 bugs doesn't mean you should go open up 40 bugs in firefox's bug tracker and paste in everything
(02:35:11 PM) jcastro: Instead you should use it as a guide to get started with
(02:35:29 PM) jcastro: You don't want to be in a vacuum forwarding these bugs without working with someone
(02:35:39 PM) jcastro: so for example for these bugs I would look at them
(02:35:47 PM) jcastro: determine which ones would be easy low hanging fruit
(02:36:01 PM) jcastro: then I would post on the ubuntu-mozilla mailing list or ask someone in the irc channel
(02:36:35 PM) jcastro: something like "I am interested in helping triage bugs upstream, I am looking at this list here, any thoughts?"
(02:36:38 PM) jcastro: or something like that
(02:37:07 PM) jcastro: at which point someone on the team would give you some advice or tell you what to do or point you to more information
(02:37:19 PM) jcastro: It is important to tell someone in a team that you are looking at those bugs
(02:37:45 PM) jcastro: for example the desktop team does a great job of doing linkages, they have their little way to handle upstreaming bugs
(02:38:05 PM) jcastro: so I try not to touch them unless it is obvious that I can do it
(02:38:13 PM) jcastro: even then, sometimes I make a mistake and link the wrong bug
(02:38:32 PM) jcastro: at which case someone will tell you it's a mistake and then fix it
(02:38:53 PM) jcastro: Any more questions so far?
(02:39:13 PM) jcastro: ok so more tips
(02:39:32 PM) jcastro: So it's obvious that bugs filed under "linux" are far and away the most open bugs.
(02:39:54 PM) jcastro: These are tough because in a lot of cases they are hardware specific
(02:39:58 PM) jcastro: and not very easy to track down
(02:40:13 PM) jcastro: so in the case of kernel bugs you'll definately want to work with the kernel team on that
(02:40:32 PM) jcastro: Sometimes you look at an ubuntu bug
(02:40:44 PM) jcastro: and then you find a bug in the upstream project that might be similar
(02:40:47 PM) jcastro: but you might not know
(02:41:04 PM) jcastro: there is absolutely nothing wrong with posting a comment asking "Is this the same bug as this one?"
(02:41:14 PM) jcastro: I'm sure many of you have seen these kinds of comments
(02:41:37 PM) jcastro: it's basically people who /think/ they might have found the same bug, but aren't sure, so they're asking for help from someone who might know
(02:41:47 PM) jcastro: there's nothing wrong with that of course!
(02:42:19 PM) jcastro: Ok, so the last thing before I open it up to more  questions
(02:42:27 PM) jcastro: https://launchpad.net/ubuntu/+upstreamreport
(02:42:33 PM) jcastro: If you scroll to the very bottom
(02:42:44 PM) jcastro: you'll see a total for the top100 projects
(02:43:05 PM) jcastro: the two that I look at is the percentage of bugs that are determined to be upstreamed, and the amount of linkages
(02:43:16 PM) jcastro: those numbers are 20.12% and 93.68%
(02:43:29 PM) jcastro: this tells us that out of all the  bugs we have reported
(02:43:36 PM) jcastro: 20% are determined to be upstreamed
(02:43:45 PM) jcastro: and of those, 93.68% are linked
(02:44:07 PM) jcastro: The 93% tells us that when we do find that a bug is upstreamable, that we do a great job at making that link
(02:44:20 PM) jcastro: that's people out there finding the bug in an upstream tracker and linking it in launchpad.
(02:44:28 PM) jcastro: I am interested in growing the first number.
(02:44:54 PM) jcastro: The more we determine that bugs are upstream the larger our targets for linkages become
(02:45:02 PM) jcastro: One last example
(02:45:08 PM) jcastro: look at virt-manager in this chart
(02:45:12 PM) jcastro: it's red for a reason. :D
(02:45:36 PM) jcastro: Out of 21 possible bugs that can be upstreamed, 0 have been reported to the virt-manager developers
(02:45:47 PM) jcastro: that's where we can help out to get those bugs to the right people.
(02:46:00 PM) jcastro: More questions?
(02:46:26 PM) jcastro: Ok well, I realized I chatted a bunch and it's a lot to swallow
(02:46:39 PM) jcastro: but hopefully you will be able to use the report to find bugs that can be linked
(02:46:54 PM) jcastro: if anyone has any feedback or questions on forwarding reports, please let me know
(02:47:04 PM) jcastro: I am always on irc and can be reached via jorge@ubuntu.com
(02:50:46 PM) jcastro: QUESTION: Are you planning to extend the "upstreamed" list for more packages?
(02:50:49 PM) jcastro: yes
(02:51:01 PM) jcastro: currently we went for the top100 by open bugs because that covers a huge area
(02:51:11 PM) jcastro: eventually you'll be able to just ask for a package
(02:51:14 PM) jcastro: and get all that info
(02:52:06 PM) JontheEchidna: in fact I think that the top 100 packages constitute about half of all open bugs in Launchpad
(02:52:30 PM) jcastro: yes
(02:52:36 PM) jcastro: I got another question
(02:52:38 PM) jcastro: let me paste it
(02:52:43 PM) jcastro: "Hi I got a question. You said that one should not touch all bugs for a package in one go for example. I did not get the reason  behind this and my other question is what can be done to increase the reporting in upstream trackers both automagicaly and\or  with human labour do you think?"
(02:53:09 PM) jcastro: ok, so the reason we don't do this automatically is that we depend on your human brain to do a common sense thing
(02:53:18 PM) jcastro: sometimes the bug might be incorrectly marked as upstream
(02:53:39 PM) jcastro: or sometimes a developer won't mark a bug as upstreamed until he/she gets more information
(02:53:51 PM) jcastro: what we don't want is to flood upstream bug trackers with bad bugs
(02:54:28 PM) jcastro: ideally when an upstream gets a link from us in their bug tracker they should (hopefully) think "Ah, another one of those ubuntu studs with a rocking bug report"
(02:54:58 PM) jcastro: This is why it's important to ask someone in a team working on that package for guidance
(02:55:29 PM) jcastro: Because as it turns out, if you're forwarding bad bugs upstream that shouldn't be, you end up doubling the work for the ubuntu developer, the upstream developer, and probably everyone subscribed to the bug
(02:55:45 PM) JontheEchidna: they especially don't like it when it turns out to be an ubuntu bug :P
(02:55:50 PM) jcastro: So it's important that you try to do the  right thing
(02:56:00 PM) jcastro: instead of fast-fooding a bunch of bad bugs
(02:56:13 PM) jcastro: that being said, if you make a mistake, just fix it and move on
(02:56:36 PM) jcastro: I mislinked a bug this morning and found out right away that I had made a mistake. :D
(02:56:49 PM) jcastro: Question: So what can we do to make it more effective_
(02:56:55 PM) jcastro: So really alot of this is experience
(02:57:04 PM) jcastro: I recommened participating in hug days
(02:57:16 PM) jcastro: where you can join a channel and during that entire day you are focused on one package
(02:57:27 PM) jcastro: our bugmaster and QA people are around for these
(02:57:36 PM) jcastro: so those are the days to ask questions and pick their brains
(02:57:47 PM) jcastro: after a while you will just start to get good at it, like any skill
(02:57:58 PM) jcastro: ok so I am out of time, thanks everyone