20111221
Meeting summary
- Previous Actions
ACTION: patrickmw to publish a list of launchpad projects that conform our automated testing in jenkins (gema, 17:04:01)
- Blueprints Update Precise
https://blueprints.launchpad.net/ubuntu/+spec/other-p-builds-smoke-testing (gema, 17:06:49)
#subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-boot-speed-testing (gema, 17:12:44)
https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-boot-speed-testing (gema, 17:13:08)
https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-kernel-sru (gema, 17:14:36)
https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-test-case-management-tool (gema, 17:14:59)
https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-backlog (gema, 17:39:19)
https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-qa-regression-testing (gema, 17:42:25)
https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-automated-test-submissions (gema, 17:44:45)
https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-metrics (gema, 17:45:12)
https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-standard-sru-testing (gema, 17:45:26)
Community Tasks - https://wiki.ubuntu.com/QATeam/TasksPrecise (gema, 17:45:45)
LINK: https://launchpad.net/~nuclearbob/+archive/ppa (nuclearbob, 17:43:20)
- Update Lubuntu
- Update Xubuntu
- Update Ubuntu
- Other Topics
Meeting ended at 17:57:08 UTC.
Action items
- patrickmw to publish a list of launchpad projects that conform our automated testing in jenkins
People present (lines said)
- gema (173)
- alourie (93)
- roignac (15)
- nuclearbob (12)
- ScottK (12)
- phillw (9)
- meetingology (4)
- balloons (4)
Full Log
17:02:01 <gema> #startmeeting QA Meeting
17:02:01 <meetingology> Meeting started Wed Dec 21 17:02:01 2011 UTC. The chair is gema. Information about MeetBot at http://wiki.ubuntu.com/AlanBell/mootbot.
17:02:01 <meetingology>
17:02:01 <meetingology> Available commands: #accept #accepted #action #agree #agreed #chair #commands #endmeeting #endvote #halp #help #idea #info #link #lurk #meetingname #meetingtopic #nick #progress #rejected #replay #restrictlogs #save #startmeeting #subtopic #topic #unchair #undo #unlurk #vote #voters #votesrequired
17:02:10 <gema> who is with us today?
17:02:14 <alourie> o/
17:02:15 <nuclearbob> I am
17:02:23 <phillw> o/
17:02:39 <alourie> I'll try to ping roignac
17:02:44 <gema> ok, it sounds like we have many people already on holidays
17:03:04 <gema> let's get started then
17:03:08 <gema> #topic Previous Actions
17:03:38 <gema> we had an action on patrick but he is not available today, so I will postpone it again!
17:04:01 <gema> #action patrickmw to publish a list of launchpad projects that conform our automated testing in jenkins 17:04:01 * meetingology patrickmw to publish a list of launchpad projects that conform our automated testing in jenkins 17:04:11 * gema gema to talk to hggdh about some jenkins training for the community
17:04:35 <gema> I spoke to hggdh and he has organised a training session in one developer event that is happening past mid January
17:04:55 <gema> I have asked him to send an email to the list but he was going on holidays, so I guess that'll happen next year 17:05:10 * gema gema to move the standard sru testing blueprint to the end
17:05:17 <alourie> yea!!
17:05:22 <gema> I did that on the agenda
17:05:53 <gema> anything I am missing, anyone?
17:06:00 <alourie> if we could contribute to anything jenkins oriented, would be great
17:06:16 <gema> yep, let me get to that later
17:06:21 <alourie> k
17:06:30 <gema> #topic Blueprints Update Precise
17:06:49 <gema> #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-builds-smoke-testing
17:06:59 <gema> I have started reviewing the code of our ISO smoke tests and will have some feedback early in January.
17:07:16 <gema> I am not sure who to send the feedback to, but will figure it out
17:07:31 <alourie> o/
17:07:40 <gema> alourie: go ahead
17:07:56 <alourie> I think that generic stuff like that should go to the list anyway
17:08:03 <alourie> s/list/maillist
17:08:18 <gema> yes, I can copy the list about it, but I don't know who is responsible to fix it
17:08:21 <alourie> and then other channels if necessary
17:08:21 <alourie> ..
17:08:23 <alourie> ah
17:08:25 <alourie> ok
17:08:44 <gema> I will definitely copy ubuntu-qa whenever I send it
17:08:50 <alourie> great
17:08:52 <alourie> thank you
17:08:59 <gema> no problem
17:09:02 <roignac> o/
17:09:07 <gema> roignac: go ahead
17:09:24 <roignac> you should also include ubuntu-server-iso-testing-dev@lists.launchpad.net
17:09:43 <roignac> this is a maillist of Ubuntu Server Iso Testing Developers
17:09:44 <gema> ahh, so those are the people responsible for it, you reckon?
17:09:48 <gema> ok
17:09:53 <gema> thanks roignac
17:10:17 <roignac> i guess so, please include other drivers/maintainers of automated test project (based on info from patrickw)
17:10:30 <roignac> s/patrickw/patrickmw/
17:10:35 <gema> yes, will do
17:10:56 <gema> I am also thinking that they may tell me to fix it myself, I am not sure how these things work
17:11:01 <gema> but I guess we'll find out
17:11:28 <gema> we are also looking into modifying the jenkins plugin we installed for the ISO testing dashboard
17:11:47 <gema> so that it shows the build we are testing, aggregates results, etc
17:12:09 <gema> we'll let you know if we go ahead with the enhancements, the plan would be to contribute those upstream
17:12:44 <gema> #subtopic #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-boot-speed-testing
17:12:57 <gema> too many subtopics, let me try again
17:13:08 <gema> #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-boot-speed-testing
17:13:27 <gema> patrickmw is not here, nuclearbob do you have any update on this?
17:13:40 <nuclearbob> yeah, we're working on adding architecture to the results now
17:13:52 <nuclearbob> as part of that we're redoing the directory structure
17:14:05 <nuclearbob> and we'll be adding some information about deltas on a given day and between days
17:14:17 <nuclearbob> that's all I've got for now
17:14:24 <gema> cool, thanks
17:14:36 <gema> #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-kernel-sru
17:14:48 <gema> sconklin is not here either
17:14:49 <gema> moving on
17:14:59 <gema> #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-test-case-management-tool
17:15:16 <gema> the Mozilla team plan to have the new version of Case Conductor ready for mid January
17:15:38 <roignac> o/
17:15:41 <gema> they answered yesterday night to my email and are happy to discuss our requests at some point in January
17:15:45 <gema> yes, roignac ?
17:16:09 <roignac> could you please ask them whether we could use cc.oddsites.net and cc-dev.oddsotes.net freely?
17:16:22 <roignac> As I've taken liberty to create a Ubiquity product
17:16:26 <gema> I did, they are happy for us to perform our tasks there
17:16:34 <gema> as long as we don't destroy their testing envs
17:16:38 <gema> since they are also using it
17:16:41 <roignac> and later realised that this could break some of their automated tests
17:16:52 <roignac> oh, ok. No destruction anyway
17:16:53 <gema> no, I think they run their automated tests in other place
17:16:58 <gema> cool
17:17:03 <roignac> By the way
17:17:17 <alourie> o/
17:17:17 <roignac> I'd like someone to try review feature in CC
17:17:30 <gema> roignac: which feature?
17:17:44 <gema> alourie, you can speak freely
17:17:48 <alourie> ok
17:17:59 <roignac> Case Conductor allows us to review testcases - though I haven't figured how for now
17:18:17 <roignac> So we could review new testcases there, how about that?
17:18:18 <gema> roignac: I don't think it does, we asked for them to add the under review state
17:18:27 <alourie> From what I've seen , CC also offers flow of testing.
17:18:41 <gema> roignac: that is a test environment, we shouldn't be using it to store data
17:18:48 <gema> it can be teared apart any moment
17:18:49 <alourie> we use ISO tracker for that (well, in context of testing ISO images, at least)
17:19:03 <alourie> so what is our purpose with CC? Is it only for storing test cases?
17:19:17 <gema> alourie: storing test cases and results, I would say
17:19:28 <gema> if it shows them in a meaningful way
17:19:28 <roignac> alourie, it has some reports too
17:19:31 <alourie> so, I would use it for reporting as well?
17:19:38 <gema> I think so
17:19:38 <alourie> instead of ISO tracker?
17:19:42 <gema> eventually
17:19:46 <alourie> Ah...
17:19:48 <gema> but that's not going to happen any time soon
17:19:55 <gema> it's going to take us a while to get there
17:19:59 <alourie> hm, ok
17:20:06 <gema> and if it is not better than the iso tracker, then we won't use it for that
17:20:14 <alourie> got ya
17:20:27 <alourie> so it could only be used for storing test cases
17:20:33 <gema> to start with, yes
17:20:48 <alourie> and if it does good job with it, we can only use it for storing
17:20:51 <gema> and for community testing to report results
17:20:55 <gema> and defects
17:21:04 <gema> we just need to find a way to connect it to the tracker database
17:21:05 <alourie> gema: but that's what ISO tracker is for!
17:21:10 <alourie> hm
17:21:10 <gema> which doesn't seem very difficult
17:21:18 <balloons> hello everyone -- late, but thought I would lurk
17:21:24 <gema> balloons: welcome!
17:21:47 <alourie> gema: ok, I'm not clear on this, but let's just waut
17:21:54 <gema> alourie: there are two functionalities on the tracker, one is to report results, the other one is to display them
17:22:06 <gema> alourie: on a first step we are trying to replace the wiki
17:22:14 <alourie> sure
17:22:15 <gema> which is horrible to store test cases
17:22:24 <alourie> +1
17:22:28 <gema> after that, we'll see if we use the tracker fully, or just partially
17:22:35 <gema> depending on how CC integrates with our workflow
17:22:43 <gema> but this is not going to happen overnight in any case
17:22:50 <alourie> I think that test content should show on the flow, either tracker or CC
17:23:01 <alourie> we'll see how CC progresses, and make our minds later.
17:23:06 <gema> and we don't want to screw up any release so we will do a gradual move to CC
17:23:15 <gema> alourie: agreed
17:23:33 <alourie> hey, if they can handle 'test case management', this is already an improvement
17:23:38 <gema> yes
17:23:41 <alourie> even without flows
17:23:48 <gema> it'll flow, just give it time
17:23:55 <alourie> sure it will
17:24:03 <gema> and this will probably free some of our good devs to do some more interesting stuff
17:24:22 <gema> ok, the other thing I wanted to say on this topic
17:24:24 <alourie> roignac: if you find that review option, ping me and we'll schedule something
17:24:36 <gema> I have been thinking about how to classify the test cases and how to structure the testing, I am not sure whether we should be testing images or whether our
17:24:45 <gema> test cases should start targetting packages themselves.
17:24:59 <alourie> o/
17:25:04 <gema> alourie: go ahead
17:25:15 <alourie> I think these are different kinds of testing
17:25:25 <alourie> you test images for being a complete product
17:25:42 <gema> that's the thing, alourie they are not
17:25:43 <alourie> but you test packages for verifying that they work
17:25:47 <alourie> gema: why not?
17:26:11 <gema> alourie: because two different people choose two different options at install time and end up with two different set of packages installed
17:26:25 <gema> so there is no one product in an image
17:26:33 <gema> there is virtually infite products in it
17:26:45 <alourie> hm
17:27:25 <alourie> so what would be purpose of ISO testing?
17:27:25 <gema> so I guess the question is whether we are testing the end product or an artificial construct we call default image
17:27:45 <gema> testing as many flavors of the ISO as we can
17:27:51 <gema> taking them into account
17:27:55 <alourie> gema: would it be too inaccurate to assume that most people install the default image?
17:27:56 <gema> unlike now
17:28:05 <gema> alourie: I believe so
17:28:14 <alourie> so testing the default flow makes sense
17:28:30 <gema> yes, and testing others makes equal sense
17:28:51 <ScottK> The question I'd ask is "what are you testing".
17:29:04 <ScottK> Generally for ISO testing you're testing the image and the installer.
17:29:15 <ScottK> You aren't really testing all the applications.
17:29:23 <alourie> so, we are actually questioning ourselves with "what purpose does ISO testing have?"
17:29:25 <ScottK> That can allow you to narrow your test focus.
17:29:37 <alourie> I'd agree with ScottK on this
17:30:08 <ScottK> People who are running the development release need to test does the stuff work after it's installed and running.
17:30:38 <phillw> indeed, getting it installed and running is an important part
17:31:14 <gema> agreed, so what would you propose we do, ScottK ?
17:32:03 <ScottK> I think the test cases we've had are appropriately focused on what ISO testing needs to accomplish.
17:32:21 <ScottK> That doesn't mean they can't be improved, but they are at least in the ballpark.
17:32:43 <ScottK> What I've seen though is that as the installer evolves, the test cases don't keep up.
17:33:03 <gema> that's the thing, I don't think they are appropriate
17:33:22 <gema> because they are mainly installer tests, not ISO tests
17:33:30 <balloons> ScottK: sounds like the classic issue of adding a feature (code-wise), but not docs or tests, etc
17:33:36 <gema> they don't tend to verify much of what's actually installed works
17:34:12 <alourie> gema: so do we touch here definition of "ISO" as the complete system?
17:34:34 <alourie> so, by "ISO testing" we imply testing that EVERYTHING works...
17:34:42 <gema> I would have thought so
17:34:44 <alourie> or, rather, "tested"
17:34:48 <alourie> hm
17:34:55 <ScottK> Then you better set aside a month or two for testing in the release cycle.
17:35:08 <alourie> that "EVERYTHING" bit worries me a little
17:35:13 <gema> ScottK: or get more people collaborating and get as much as you can automated
17:35:23 <phillw> yikes, test every default app that the standard iso holds?
17:35:25 <ScottK> We're an integrator and have to, to some extent, rely on upstreams to deliver working code.
17:35:30 <gema> phillw: yes
17:35:39 <alourie> wow
17:35:42 <phillw> this would take some time!
17:35:47 <gema> ScottK: we are an integrator and have to verify things integrate properly
17:35:48 <alourie> I could see a problem here
17:36:18 <ScottK> gema: True, but not all of that verification needs to be done in the context of ISO testing.
17:36:18 <gema> well, this is a bigger problem than this meeting, just wanted to let you know I have started to think about it
17:36:20 <alourie> is it even viable?
17:36:26 <gema> ScottK: agreed
17:36:42 <gema> ScottK: but it needs to be done by ISO release time
17:36:50 <gema> alourie: it is
17:36:53 <alourie> gema: so here's an idea
17:36:55 <ScottK> It needs to be done.
17:37:00 <gema> ok
17:37:02 <alourie> what if we split this into 2:
17:37:13 <alourie> 1. ISO testing part 1, which includes installer and image
17:37:26 <alourie> 2. ISO testing part 2, which includes testing that apps work
17:37:27 <alourie> ..
17:37:46 <alourie> in general, they could be done in parallel/separately
17:37:55 <gema> alourie: the apps part we need to automate, I don't think running it manually unless it is necessary due to the nature of the test case works
17:38:21 <gema> alourie: I don't see how you can test that the apps work without the installer working properly
17:38:31 <gema> cos you don't have the system under test installed in that case
17:38:39 <gema> anyway, we can keep discussing this on the list
17:38:48 <alourie> ok, let's take it offline
17:38:56 <gema> it was just a thought that occurred to me whilst trying to figure out how to populate CC
17:39:03 <alourie> gema: ok
17:39:12 <alourie> gema: I can also tell you about my secret idea then
17:39:16 <alourie>
17:39:19 <gema> #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-backlog
17:39:22 <gema> alourie: cool
17:39:30 <gema> any backlog info to share today?
17:39:54 <alourie> are community tasks separate item nowdays?
17:40:08 <gema> they are in case something got lost
17:40:12 <gema> but we can discuss here too
17:40:12 <alourie> ok
17:40:21 <alourie> so I can go with wiki updates
17:40:23 <gema> I haven't added the tasks to the blueprints yet because I didn't find the time
17:40:27 <gema> but you can go ahead, yes
17:40:49 <alourie> ok, so I didn't have much time to work on wiki this week, but I did make some progress
17:41:10 <alourie> I will also make a hierarchy of the thing I want to update, and work on them too
17:41:22 <alourie> so, I hope that I'll have something to update with soon
17:41:30 <alourie> and then I'll send it to the list for review
17:41:31 <alourie> ..
17:41:57 <gema> alourie: excellent, thanks a lot
17:42:11 <gema> anything else anyone
17:42:12 <gema> ?
17:42:25 <gema> #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-qa-regression-testing
17:42:28 <gema> nuclearbob: ?
17:42:44 <nuclearbob> I've got autotest packaged now, autotest-server and autotest-client are in my ppa
17:43:00 <gema> could you post the launchpad link to it?
17:43:01 <nuclearbob> I've been using those to run tests on the most recent images available in our openstack cluster
17:43:03 <nuclearbob> yeah
17:43:20 <nuclearbob> https://launchpad.net/~nuclearbob/+archive/ppa
17:43:22 <gema> thanks
17:43:55 <gema> ok, thanks a lot nuclearbob
17:44:10 <gema> #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-automated-test-submissions
17:44:12 <nuclearbob> some of the tests are failing on the precise dailies, so I'm doing more investigation to determine whether the failures are actual problems or whether the test code needs to be updated
17:44:15 <nuclearbob> looks like some of both
17:44:16 <gema> ups, sorry
17:44:18 <nuclearbob> that's all from me
17:44:35 <gema> ok, sounds good, thanks
17:44:45 <gema> #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-automated-test-submissions
17:45:04 <gema> no progress on this task this week, so nothing to report
17:45:12 <gema> #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-metrics
17:45:17 <gema> no progress on this one either
17:45:26 <gema> #subtopic https://blueprints.launchpad.net/ubuntu/+spec/other-p-qa-standard-sru-testing
17:45:37 <gema> we don't have jibel around, so no progress on this one either
17:45:45 <gema> #subtopic Community Tasks - https://wiki.ubuntu.com/QATeam/TasksPrecise
17:45:53 <gema> have we missed any community task ?
17:46:06 <gema> that we have people here to discuss, I mean
17:46:10 <alourie> I don't think so
17:46:14 <gema> ok, so moving on
17:46:28 <gema> #topic Update Lubuntu
17:46:33 <gema> phillw: ?
17:46:47 <phillw> all is peaceful, no horrible show-stopping bugs reported.
17:47:01 <gema> good
17:47:18 <gema> anything else, phillw ?
17:47:41 <phillw> at the meeting tonight they are looking to form some teams... I'm hoping for a QA / testing team
17:47:57 <gema> that'd be good, yes
17:48:08 <alourie> phillw, gema: would you elaborate a bit?
17:48:21 <alourie> a, the Lubuntu
17:48:44 <gema> I guess phillw is getting himself a QA Team for Lubuntu, yes
17:49:00 <alourie> gema, phillw: this reminds me
17:49:05 <phillw> alourie: currently everything lubuntu is on one mailing list, we are looking to start to split groups so, for example, normal users do not get the chatter from the devs about what they're working on
17:49:23 <alourie> phillw: sure,
17:49:46 <alourie> currently, when we list different QA subgroups on wiki, we list Kubuntu QA
17:50:10 <alourie> so, I think we need to list all the QA that we have in -buntu universe there
17:50:17 <phillw> alourie: that would be correct, as I'm the only person on the Lubuntu QA team (and I get lonely :P )
17:50:23 <alourie>
17:50:49 <gema> good, alourie will you take care of that?
17:50:52 <gema> we are running out of time
17:50:54 <alourie> gema: of course
17:51:10 <gema> cool, thanks
17:51:14 <gema> #topic Update Xubuntu
17:51:24 <gema> charlie doesn't seem to be around
17:51:28 <gema> so we are moving on
17:51:47 <gema> #topic Update Ubuntu
17:52:01 <gema> I don't think we have any updates for Ubuntu either, besides what we've already discussed
17:52:10 <gema> #topic Other Topics
17:52:23 <gema> I'd like to mention that we are not having our weekly meeting next week
17:52:28 <gema> due to holidays and festivities
17:52:34 <gema> are we all happy with that?
17:52:41 <gema> or do you guys want to have it?
17:52:41 <alourie> sure
17:52:47 <phillw> okay with me.
17:52:53 <alourie> ok here
17:53:05 <gema> ok, so our next meeting will be the 4th of January
17:53:32 <gema> if anyone wants to chair it, let me know, otherwise I will do it
17:53:47 <gema> any other topic ?
17:53:50 <alourie> gema: what was that bit about "participating in jenkins" stuff, that you wanted to mention later?
17:54:04 <gema> ahh, we'll start with the training that hggdh is going to give
17:54:14 <gema> and then we could have ubuntu-qa triaging issues
17:54:28 <gema> or at least, trying to learn how to do it
17:54:31 <alourie> triaging issues?
17:54:44 <gema> as in, when a test fails, determine why and raise a defect if required
17:54:52 <alourie> aha
17:54:53 <gema> or try to reproduce on your environment to make sure it happens
17:54:56 <alourie> ok
17:54:56 <gema> that sort of thing
17:55:05 <alourie> sure
17:55:05 <gema> but we need some training for that
17:55:16 <alourie> nod 17:55:23 * alourie loves this stuff
17:55:27 <balloons> does the bugsquad have anything for it now?
17:55:42 <gema> no, bugsquad are doing different things
17:55:47 <gema> afaik
17:56:19 <gema> we are doing it within canonical QA
17:56:31 <gema> but there is no reason why the community contributors cannot do it if they like to
17:56:38 <balloons> right right..
17:56:56 <gema> so that'd be it, any further qustions, please send to the list!
17:56:59 <alourie> balloons: we split qa and bugsquad to 2 teams
17:57:08 <gema> #endmeeting
Generated by MeetBot 0.1.5 (http://wiki.ubuntu.com/AlanBell/mootbot)
QATeam/Meetings/QA/20111221 (last edited 2011-12-21 18:01:36 by 94-192-45-141)