#title #ubuntu-meeting: Ubuntu QA
Please note this meeting is on two parts with the continuation further down the page.
Meeting started by balloons at 17:03:25 UTC. The full logs are available at http://ubottu.com/meetingology/logs/ubuntu-meeting/2012/ubuntu-meeting.2012-02-29-17.03.log.html .
Meeting summary 1st part
- Ubuntu Updates
LINK: http://iso.qa.ubuntu.com/qatracker/milestones/208/builds/12727/testcases (brendand, 17:12:23) LINK: http://launchpad.net/bugs/942573 (brendand, 17:12:34)
- Ubuntu Flavor Updates
LINK: http://iso.qa.ubuntu.com/qatracker/reports/bugs/938472 (phillw, 17:17:37) LINK: https://code.launchpad.net/~ubuntu-server-iso-testing-dev/ubuntu-server-iso-testing/trunk (jibel, 17:26:35)
- Other topics
Meeting ended at 18:00:19 UTC.
People present (lines said)
- balloons (91)
- brendand (46)
- jibel (19)
- phillw (16)
- ubottu (5)
- Riddell (4)
- meetingology (3)
- josepht (1)
17:03:25 <balloons> #startmeeting Ubuntu QA
17:03:25 <meetingology> Available commands: #accept #accepted #action #agree #agreed #chair #commands #endmeeting #endvote #halp #help #idea #info #link #lurk #meetingname #meetingtopic #nick #progress #rejected #replay #restrictlogs #save #startmeeting #subtopic #topic #unchair #undo #unlurk #vote #voters #votesrequired
17:03:39 <balloons> can we get a roll call of who's around
17:03:54 <phillw> o/
17:05:08 <phillw> could be a short meeting :/
17:05:22 <balloons> heh
17:05:24 <brendand> o/
17:05:32 <josepht> o/
17:06:00 <balloons> alrighty.. well let's dive into the agenda
17:06:12 <balloons> there was one outstanding item, for me to update the wiki 17:06:23 * balloons balloons to update QA Team wiki heading
17:06:31 <balloons> i wanted to add an activities icon
17:06:39 <balloons> I seem to only be successful at breaking the header page
17:07:50 <phillw> cproffit from ubuntu-begginers helped me when I got stuck with the new lubuntu header
17:08:21 <balloons> phillw, awesome, I'll consult him.. expect to see that today after the meeting
17:08:32 <balloons> i picked the <3 icon for activities
17:08:33 <balloons> hehe
17:08:51 <balloons> [TOPIC] Ubuntu Updates
17:09:12 <balloons> jibel, are you around?
17:09:24 <jibel> hey
17:09:41 <balloons> hello care to update us on ubuntu qa activities this week?
17:10:26 <jibel> sure, Precise Beta 1 testing in progress
17:10:46 <jibel> list of images that need testing http://iso.qa.ubuntu.com/qatracker/milestones/208/builds
17:11:18 <jibel> a bunch of bugs has been found http://iso.qa.ubuntu.com/qatracker/reports/defects/opened
17:11:34 <jibel> major bugs with the installer and the live session are:
17:11:43 <jibel> bug 940908
17:11:45 <ubottu> Launchpad bug 940908 in ubiquity (Ubuntu Precise) "Keyboard layout not set on persistent USB image" [High,Confirmed] https://launchpad.net/bugs/940908
17:11:52 <jibel> bug 939450
17:11:53 <ubottu> Launchpad bug 939450 in ubiquity (Ubuntu Precise) "ubiquity crashed with TypeError: argument of type 'NoneType' is not iterable in ubi-partman.py" [High,Triaged] https://launchpad.net/bugs/939450
17:11:59 <jibel> bug 942560
17:12:00 <ubottu> Launchpad bug 942560 in ubiquity (Ubuntu) "keyboard layout screen - Keyboard navigation broken" [High,Triaged] https://launchpad.net/bugs/942560
17:12:11 <brendand> o/
17:12:19 <balloons> yes brendand ?
17:12:23 <jibel> and bug 942030
17:12:23 <brendand> http://iso.qa.ubuntu.com/qatracker/milestones/208/builds/12727/testcases
17:12:24 <ubottu> Launchpad bug 942030 in ubiquity (Ubuntu) "'Take a photo' preview pane not showing live preview" [High,Confirmed] https://launchpad.net/bugs/942030
17:12:34 <brendand> http://launchpad.net/bugs/942573
17:12:35 <ubottu> Launchpad bug 942573 in xubuntu-default-settings (Ubuntu) "Network connections indicator not shown" [Undecided,Fix released]
17:12:45 <brendand> is in xubuntu, but marked against an ubuntu testcase??
17:13:53 <jibel> right, I saw that but didn't talk to primes2h yet
17:15:05 <jibel> on the image testing side, a tester reported a hang on the keyboard layout screen on the second stage of an OEM install
17:15:49 <jibel> I've been unable to reproduce, it be nice if we could have more testers to validate that if it is a local issue or not
17:16:31 <balloons> jibel, can we draft some of this up into a mail for the qa list? I'll also send a tweet on status
17:16:32 <jibel> apart from that, beta 1 should be released tomorrow, that's plenty of hours for more coverage
17:16:42 <jibel> balloons, ok
17:16:50 <balloons> thanks jibel!
17:16:53 <jibel> ..
17:17:15 <balloons> Alright, let's talk about flavors now
17:17:24 <balloons> [TOPIC] Ubuntu Flavor Updates
17:17:31 <balloons> lubuntu first, how's that
17:17:37 <phillw> http://iso.qa.ubuntu.com/qatracker/reports/bugs/938472
17:18:05 <phillw> We're all over it like a rash, this was caused by Â it's a change on GTK3 3.2.1 syntax and theme standards (I'm angry because of so many changes, they won't make any standard if it goes that way) and Unico 1.0.1 (some textures changes, too).
17:18:19 <phillw> Which pulled the rug from under lubuntu for a week.
17:19:05 <phillw> a rough patch is hoped for today, and then all the isos will need respinning.
17:19:15 <balloons> yikes
17:19:54 <phillw> IMHO, it was an epic fail via QA for this to happen. We had no warning nor information. Just black screens.
17:21:13 <balloons> alrighty. good luck, I'm looking at the screenshot of the issue
17:21:27 <balloons> thata's no fun.. and very vague as to what could be the issue
17:21:30 <phillw> so, the request is - is there any way to stop this in thre future?
17:21:30 <balloons> glad you isolated it
17:21:56 <phillw> our graphics guy is proud of his work and was gutted over it
17:22:05 <balloons> that's no good at all
17:22:53 <balloons> I feel like the future for stopping this is doing more testing of key components before they entire the archive
17:22:59 <phillw> the only other thing to note is the ppc *may* not be working on G3 macs, but until we can actually install it's hard for our few testers to test
17:23:20 <balloons> I'll sidebar for a moment to mention this list: https://wiki.ubuntu.com/PrecisePangolin/DesktopCriticalPackages
17:23:27 <brendand> o/
17:23:34 <balloons> sure, go ahead brendand
17:23:45 <brendand> is lubuntu being iso tested?
17:23:49 <brendand> in jenkins?
17:24:29 <balloons> I'm not sure phillw ? jibel?
17:25:06 <phillw> balloons: I know very little about jenkins
17:26:34 <jibel> balloons, we test ubuntu, but the code we use for testing is available if you want to run it on your own infrastructure
17:27:07 <balloons> brendand, do you feel like having jenkins on this would have helped in this case?
17:27:14 <balloons> is that the answer going forward?
17:27:53 <brendand> balloons, well it looks like an integration issue to me
17:28:15 <brendand> balloons, as in Gtk didn't break itself , it broke something else
17:28:45 <balloons> k, let;s move this discussion to the mailing list.. phillw it looks like there may be a way to improve QA for future
17:28:57 <phillw> thanks.
17:29:04 <balloons> i'm sure brendand and jibel will speak more about it there
17:29:05 <jibel> balloons, in this case, for ubuntu, we don't have tests that cover the rendering of the UI and similar defects are found with manual testing
17:29:30 <balloons> jibel, thanks for the info. it certainly depends on what your coverage is
17:29:44 <balloons> ok, how about kubuntu?
17:29:46 <jibel> but the plan is to be able to do this type of test with the next testing framework we are working on
17:29:53 <brendand> phillw - i have an idea to share. on the mailing list - if you can send a message to get things rolling
17:30:20 <Riddell> brendand: looking good for beta I think
17:30:34 <Riddell> we have kubuntu-active images yet but nobody has tested and almost certainly they don't do anything
17:32:07 <balloons> yes.. any thoughts on getting hardware to try those out? will they be a published image for 12.04, or just ?
17:35:34 <balloons> Riddell, thoughts on what will be happening with the kubuntu-active images?
17:36:07 <Riddell> balloons: it'll be a "tech preview" at best
17:36:16 <balloons> ok, makes sens
17:36:19 <Riddell> if we get it going that's nice, if not then we won't worry too much
17:36:31 <balloons> awesome.. anything else?
17:37:12 <balloons> alrighty, let's move on to ubuntu studio.. anyone about care to update us?
17:38:53 <balloons> no one about..
17:39:01 <balloons> how about edubuntu?
17:40:03 <balloons> lastly, any updates from xubuntu?
17:41:39 <balloons> heh.. kk.. Let's move on
17:41:45 <balloons> we've got alot to chat about
17:42:00 <balloons> [TOPIC] Other topics
17:42:07 <brendand> o/
17:42:12 <balloons> yes brendand ?
17:42:15 <brendand> ok
17:42:17 <brendand> manual testing
17:42:27 <brendand> balloons, love what you're doing with checkbox
17:42:45 <brendand> however
17:43:00 <brendand> the whole story around creating test cases is really messy
17:43:12 <brendand> we have a wiki, but it's not complete
17:43:21 <brendand> we have a spreadsheet, i'm not sure if that's complete
17:43:28 <brendand> we're looking at case conductor
17:43:41 <brendand> and now you can also submit jobs to checkbox
17:44:00 <brendand> (the checkbox-app-testing branch to be precise)
17:44:25 <balloons> brendand, yep your correct on all those things
17:44:33 <brendand> can the wiki die anytime soon?
17:44:38 <balloons> the wiki is not the long-term solution
17:44:55 <balloons> that was discussed last year afaik, and the decision to leave it was made then
17:45:17 <balloons> if not I would propose we go ahead and formally say so now
17:45:33 <brendand> balloons, yes but then as far as i can tell you're using the test cases as written on the wiki for the checkbox tests...
17:45:52 <brendand> am i wrong?
17:46:02 <balloons> I pulled the test cases we re-wrote as part of the spreadsheet rewrite project
17:46:05 <balloons> that is correct
17:46:19 <brendand> so they are *all* in the spreadsheet?
17:46:26 <balloons> all the checkbox testcases?
17:46:28 <balloons> not all all
17:46:33 <balloons> *not at all
17:46:34 <brendand> ok
17:47:05 <balloons> if you look the wiki has only maybe the nautilus and network manager tests copied unedited
17:47:23 <balloons> i took ideas from some of the others, but by and large, the wiki is pretty old and dead
17:47:29 <brendand> can we make the wiki read-only?
17:47:40 <brendand> (maybe it is already and i'm being a dum-dum)
17:47:56 <balloons> no, it's still editable.. I'm not sure the wiki can or can't be readonly
17:48:02 <balloons> if it can I would be ok with it
17:48:18 <phillw> it can be tagged as depreciated to encourage people NOT to update it.
17:48:33 <balloons> I would say we are at that point
17:48:47 <balloons> as far as the future, we are still looking for a test case management system
17:49:08 <balloons> in my post I outlined my thoughts about that.. i'll repeat them here, and see what you all think/feel
17:49:20 <brendand> also, banners saying "THIS WIKI IS DEPRECATED - SEE <Google Docs Link>"
17:49:41 <phillw> brendand: we can do that
17:49:55 <balloons> basically we need a test case mgmt system, test case delivery system and a test case reporting system
17:50:20 <brendand> what happened with case conductor?
17:50:33 <balloons> right now, we have the wiki more or less as the mgmt system, nothing (we're trying out checkbox now) as the delivry system, and nothing (we're trying out lp and the results tracker) for the reporting system
17:51:01 <brendand> i'm biased, but checkbox is a good delivery system
17:51:14 <balloons> i have been focusing my efforts recently on getting this checkbox prototype up and running
17:51:22 <phillw> balloons: brendand we can chat on ML about a couple of systems the SII have tested out if it would help?
17:51:24 <brendand> launchpad-results may need an extra layer over it to be generally suitable
17:51:34 <balloons> and the case conductor folks have been working on pushing out the new release
17:51:39 <balloons> but we will prototype that as well
17:51:56 <balloons> phillw, yes that would be great
17:52:11 <brendand> balloons, if we wanted to start using case conductor tomorrow, what would the risks be?
17:52:12 <balloons> at this point I feel like we need a nicely intergrated workflow.. but all 3 needs must be met
17:52:24 <balloons> and I'm not biased as to what tools we use
17:52:41 <balloons> brendand, yes I agree results tracker if adopted would need more added to it
17:53:19 <balloons> brendand, if we started using cc tomorrow, no real risks other than we have to adopt it.. and at the moment we have no way of delivering those tests outside of using the web browser
17:53:27 <balloons> we may or may not like that delivery system
17:53:43 <brendand> but it's still miles ahead of a spreadsheet
17:53:59 <balloons> yes.. it is
17:54:05 <brendand> balloons, have you found out does it have an api for pulling testcases?
17:54:12 <balloons> everything in the spreadsheet is in checkbox tests now though
17:54:22 <balloons> brendand, I'm not sure on the api for pulling tests
17:54:45 <brendand> balloons, you've had meeting with the mozilla guys, right?
17:54:55 <balloons> i wasn't attempting to muddy the waters, but I know having checkbox tests does deprecate the other places we've been putting them
17:55:17 <balloons> brendand, yes I've met with the mozilla guys
17:55:44 <balloons> I intend to revisit case conductor within the next couple weeks
17:55:50 <balloons> probably next week
17:56:02 <brendand> so
17:56:13 <brendand> let's clarify all this
17:56:15 <balloons> they were supposed to have a release around early march
17:56:19 <brendand> i'm a contributor
17:56:37 <brendand> and i'm reading your page, so i go and right a testcase in checkbox and propose a merge
17:57:02 <brendand> the first thing you're going to tell me is 'can you write it up in this spreadsheet too'
17:57:04 <brendand> ?
17:57:18 <brendand> s/right/write/
17:57:34 <balloons> brendand, at this point I make no mention of updating a spreadsheet or wiki (nor have I attempted to myself)
17:57:46 <balloons> aka, the checkbox tests are standalone
17:58:37 <balloons> now, we could decide to deprecate all other tests and just go with the checkbox repo.. however I didn't want to do that, even tho short-term it means we have tests in 2 places (I was assuming the wiki would stay as-is for now as the repository of tests)
17:59:02 <balloons> the long term solution is something like case conductor, which I think we all agree with
17:59:09 <balloons> it's simply what do we do in the shortterm?
17:59:53 <brendand> i think there should be some attempt to keep the authoritative location for test cases and the execution tool in sync
17:59:59 <balloons> alright our time is expired.. I'll end the meeting.. can we continue this discussion in #ubuntu-testing?
18:00:08 <brendand> sure
18:00:14 <balloons> for those folks reading the logs we'll post to the ml a summary
18:00:16 <balloons> kk
18:00:19 <balloons> #endmeeting
Meeting started by balloons at 18:01:26 UTC. The full logs are available at http://ubottu.com/meetingology/logs/ubuntu-testing/2012/ubuntu-testing.2012-02-29-18.01.log.html .
Meeting summary 2nd Part
LINK: http://bazaar.launchpad.net/~nskaggs/checkbox/checkbox-app-testing/files/head:/jobs/ (balloons, 18:22:35)
LINK: http://testcases.qa.ubuntu.com/Applications/Nautilus (balloons, 18:39:16)
LINK: http://bazaar.launchpad.net/~nskaggs/checkbox/checkbox-app-testing/view/head:/jobs/nautilus.txt.in (balloons, 18:39:23)
Meeting ended at 18:43:18 UTC.
People present (lines said)
- balloons (58)
- brendand (16)
- phillw (12)
- meetingology (3)
18:01:26 <balloons> #startmeeting ubuntu qa part II
18:01:27 <meetingology> Available commands: #accept #accepted #action #agree #agreed #chair #commands #endmeeting #endvote #halp #help #idea #info #link #lurk #meetingname #meetingtopic #nick #progress #rejected #replay #restrictlogs #save #startmeeting #subtopic #topic #unchair #undo #unlurk #vote #voters #votesrequired
18:01:42 <balloons> hehe, it's better than not capturing it, even if there will be noise
18:02:25 <balloons> alright, so yes short term, what do we do.. my thoughts on what to do with the checkbox tests after we had a test management tool was to migrate them at that point
18:02:47 <balloons> and if we still had / used the wiki (or whatever, spreadsheet, whatever) they would also be migrated
18:03:07 <balloons> our future delivery system should simply use the tests from the management system.. so they are maintained in one place
18:03:10 <brendand> balloons, don't you think it would be better if you *just* ask people to add to the spreadsheet and then you and a few other people deal with updating checkbox?
18:03:20 <balloons> and by having a nice test case management system everyone can see and update them easily
18:03:27 <balloons> without having to be a dev, or use bzr
18:04:14 <balloons> brendand, for anyone who can propose a merge, it makes it easier on me to accept the request.. if you can't I am pointing folks to adding the case to the wiki and I will manually import it
18:05:57 <brendand> ok. let's make an action item to clarify that the wiki should not be used except by those who know they need to use it
18:06:02 <balloons> for example, roignac has sent several merge requests that made it really nice on me for just accepting and not having to re-work his test cases
18:06:20 <balloons> brendand, I'm also concerned about the spreadsheet
18:06:37 <balloons> what does everyone think we should do in the short term for holding our test cases
18:06:46 <balloons> put aside the issue of how to contribute for a moment
18:08:15 <brendand> balloons, you're concerned as in you don't like it?
18:08:43 <balloons> brendand, I think I like the spreadsheet even less than the wiki, simply because it's confusing for people
18:08:58 <balloons> and everything in the spreadsheet is supposed to be/ should be in the wiki
18:09:28 <brendand> it's something of a least worst option situation
18:10:48 <balloons> for example, we could deprecate the wiki, and say checkbox branch has our testcases in it. if you want to contribute a new test and can't we could have an open spreadsheet to do so, we would manually import it, then remove the spreadsheet
18:11:03 <balloons> my concerns with that would be making it hard to see all the testcases for everyone
18:11:13 <balloons> and making it hard for the community to edit testcases
18:11:19 <balloons> however it would be a short-term thing
18:11:49 <brendand> right. i think a spreadsheet is better than a wiki for reviewing
18:12:36 <balloons> does anyone have an alternative option to present?
18:12:59 <brendand> i seem to recall that one of the reasons we moved away from the wiki was not because of it being a wiki but because many of the test cases themselves were actually useless
18:13:04 <balloons> I'm not necessarily pushing the option above, just that it was an option
18:13:19 <balloons> brendand, yes the wiki was in worse shape that I thought
18:13:32 <balloons> anything not written recently was old and useless
18:13:37 <balloons> in many cases it was blank
18:14:36 <brendand> i think we need to head toward the optimum solution post-haste
18:15:20 <brendand> make checkbox the definitive store until case-conductor is in place
18:15:50 <brendand> for those who don't want to use bzr we have a 'new test cases' sheet as balloons suggested
18:16:09 <brendand> this is reviewed every QA meeting and synced to checkbox, then wiped
18:16:31 <balloons> if there's no other ideas put forward I'm ok with doing that
18:17:21 <brendand> once case-conductor is in place AND we have a tool which syncs down test cases to checkbox, we can start putting test cases in there
18:17:55 <balloons> sounds fine to me
18:18:21 <balloons> do we feel a spreadsheet is necessary?
18:18:44 <balloons> would the mailing list suffice? or is it ok to simply ask them to bzr branch and propose a merge?
18:19:16 <brendand> balloons, it might limit the scope of contribution, but maybe for the short term
18:19:23 <brendand> ML could work
18:19:36 <brendand> i was just thinking to have an outlet for those with no bzr experience
18:20:20 <brendand> balloons, and there's still the question of the testcases.qa wiki
18:20:22 <balloons> yes I'm concerned about that
18:20:27 <balloons> i don't want to limit
18:20:35 <balloons> but perhaps just sending a mail would be best 18:21:34 * phillw has no bzr knowledge :/
18:21:52 <balloons> awesome.. so phillw what makes sense for you
18:22:05 <balloons> would you feel put off by having the tests in a bzr branch?
18:22:35 <balloons> for instance, you can see all the tests right now by browsing them online
18:22:46 <phillw> balloons: it is a case of learning and then teaching new testers. My concern is that if it is too complicated it will put them off.
18:23:53 <balloons> phillw, what would you propose then?
18:25:17 <phillw> balloons: the most difficult thing of all. A clear, easy to follow set of instructions for newcomers (And, if I can follow them, any one can.... trust me on that one when I made pcman write up how to set up debug in pcmanfm :P )
18:25:59 <balloons> phillw, +1. we are working towards that
18:26:21 <balloons> and this decision factors in.. we want this to get iteratively easier, not harder
18:26:32 <phillw> I was also the 'idiot' who idiot checked the GRUB 2 instructions way back when drs305 was writing them
18:26:35 <balloons> but the biggest piece behind this checkbox thing was to get more testers out there
18:26:41 <balloons> and I think / hope it will
18:27:21 <balloons> this problem is sort of a side issue.. the main issue I was trying to solve was to get more testing, and better metrics on testing.. in addition, I wanted to increase the number of real tests
18:27:28 <balloons> and get the tests we wrote used
18:30:07 <phillw> I can only say from personal experience, that the likes of http://testcases.qa.ubuntu.com/Plans/LaptopTesting#System are followable?
18:33:42 <brendand> balloons, i need to leave the conversation now. look forward to a few merges into your checkbox branch from me
18:35:25 <balloons> brendand, awesome thanks
18:35:44 <balloons> phillw, are you trying to say you can or can't follow that wiki page? 18:35:56 * balloons is confused on sarcasm or not
18:36:05 <phillw> balloons: I thought the wiki is due to go?
18:36:20 <balloons> phillw, long term yes
18:36:30 <balloons> but short term.. well we're deciding that now
18:37:06 <phillw> balloons: do you have the link to the wiki, please?
18:37:29 <balloons> link to the wiki? this? http://testcases.qa.ubuntu.com/
18:37:30 <phillw> I'm sufferring from tab overload
18:37:37 <balloons> ahh
18:38:34 <phillw> balloons: that looks familiar aka http://testcases.qa.ubuntu.com/Plans/LaptopTesting#System
18:39:05 <phillw> it is simply 'what do you expect to happen" followed by - Does it?
18:39:15 <balloons> so phillw.. here's a good example.. the nautilus tests are the same in the wiki and checkbox
18:39:16 <balloons> http://testcases.qa.ubuntu.com/Applications/Nautilus
18:39:42 <balloons> phillw, yes the format of those tests are really iffy
18:39:54 <balloons> they would need reworked to go into checkbox or case conductor, etc
18:40:42 <phillw> ooh, pink! Yeah, they are in plain english as to what to expect etc.
18:41:26 <phillw> Sorry, I've GTG - Dinner is served at 18:30 UTC on wednesdays here as I have the lubuntu meeting at 20:00 UTC.
18:42:45 <balloons> gotcha
18:42:47 <balloons> ok, well
18:43:06 <balloons> I'll sumarize on the mailing list
18:43:11 <balloons> thanks for the extended discussion
18:43:18 <balloons> #endmeeting