20090923

Agenda

  • Mago pending merges
    • Blacklist/Skip (eeejay)
    • Gcalctool (jtatum)
  • Mago documentation
  • How to handle release dependent test cases - framework or branches? (jtatum)

Summary

  • Pending merges
  • Call for updating the API documention:

    • The documentation needs a hug. If you want to help, please go to the current documentation page at http://people.canonical.com/~ara/mago_doc/ and check what's left or needs an update. It was agreed during the meeting that those changes won't require reviews in order to land into trunk. So, dive in! [ACTION] ara to set up a cronjob to run epydoc automatically

  • How to maintain tests for different releases?
    • It was agreed that having a branch for release is the best solution. Right now Karmic is trunk. When it gets released, then we will branch Karmic and trunk will become Lucid.

Logs

[17:30]  * jtatum waves
[17:30]  * ara waves
[17:31]  * rmcbride waves
[17:31] <ara> anyone else for the automation meeting? jcollado? cgregan? eeejay? cr3?
[17:31]  * jcollado waves
[17:32] <davmor2> cr3: got the same thing passing it on to those in the know in form of a bug report
[17:32] <ara> OK, let's start then
[17:32] <ara> [TOPIC] Mago pending merges
[17:32] <ara> We had two pending merges this week
[17:33] <ara> * Blacklist/skip tests by eeejay
[17:33] <ara> after a general approval of the change it was merged into trunk today
[17:34] <ara> I had it documented at http://mago.ubuntu.com/Documentation/RunningTests
[17:34] <ara> have you read it guys? Am I missing or misunderstanding anything?
[17:35] <jtatum> The changes look great and are aligned with my understanding of the skip element. Very thorough!
[17:35] <rmcbride> It makes sense from here. I haven't tried it yet
[17:36] <ara> OK, let's go to the second change:
[17:36] <ara> * Gcalctool (jtatum)
[17:36] <jcollado> One moment
[17:36] <ara> jcollado, go ahead
[17:37]  * ara loves how jcollado always finds something :)
[17:37] <jcollado> I thought that example 3 also run the skipped test cases because the test suite is whitelisted explicitly
[17:38] <ara> jcollado, but that test case was skipped, not the test suite
[17:38] <ara> jcollado, so it is the case that needs to be explicitly called, isn't it?
[17:38] <jcollado> Uhm, not sure
[17:38] <jcollado> Have you run those examples?
[17:39] <ara> jcollado, most of them, that one, particularly, I think I have, but let me run it again
[17:39] <jcollado> skip method is a SuiteData class property
[17:40] <jcollado> I could by my fault, is just that I thought that wasn't the behaviour
[17:40] <jcollado> We can verify later
[17:40] <jcollado> Let's move to the next point
[17:40] <ara> it is skipped, I just tried again
[17:40] <jcollado> Ok, thanks for the verification
[17:41] <davmor2> cr3: bug 435376 for your and fader_ 's viewing
[17:41] <ubot4> Launchpad bug 435376 in ubuntu "crash during install on alternate cd 20090923" [Undecided,New] https://launchpad.net/bugs/435376
[17:41] <ara> So, second change, gcalctool (jtatum)
[17:42] <ara> jcollado, made some useful comments and jtatum made the suggested changes
[17:42] <ara> jcollado, if nobody else comment, could you take the action in merging the changes?
[17:42] <jcollado> ara: Sure
[17:43] <ara> jcollado, thanks
[17:43] <ara> Next, then
[17:43] <ara> Mago documentation
[17:44] <ara> This is a call for updating the API documentation for those methods that didn't have or those not up to date
[17:44] <ara> In my opinion, those changes should be merged straight forward, without asking for merge requests
[17:44] <ara> what do you guys think?
[17:45] <mikefletcher> +1
[17:45] <rmcbride> +1
[17:45] <jcollado> I agree
[17:45] <jtatum> Looks like you already did it ;) +1
[17:45] <ara> jtatum, touché!
[17:46] <ara> API doc is here: http://people.canonical.com/~ara/mago_doc/
[17:46] <ara> If you find something in need of an update, feel free to work on it ;-)
[17:46] <jtatum> ara: do you invoke epydoc automatically?
[17:47] <ara> jtatum, I am afraid that not yet. But I will put that as an action item for myself. Thanks for the suggestion
[17:48] <ara> OK, the last item on the list:
[17:48] <ara> How to handle release dependent test cases - framework or branches? (jtatum)
[17:48] <ara> jtatum, go ahead
[17:50] <jtatum> OK. Running all the tests in Mago today is problematic. If you invoke bin/mago and just let it go, eventually you will wind up with a bunch of failed tests and error messages on the screen from both evolution and pidgin (if installed).
[17:50] <ara> very true
[17:50] <jtatum> Basically, the issue I see is that we have tests developed with versions of Ubuntu from intrepid onwards.
[17:51] <jtatum> So I was curious about the plan for this. I have a few ideas but was wondering what everyone thought? Should tests just be for the latest ubuntu and managed with bzr branches or ...?
[17:52] <mikefletcher> fyi, I ran into problems with the gnome screenshot tests because of string charges between Jaunty and Karmic.  The gnome screenshot tests would fail on Jaunty.
[17:53] <jcollado> Maybe it's time to have separate branches for test runnner and and the test cases
[17:53] <jcollado> the runner probably works find regardless the distro
[17:53] <jcollado> but the test cases maybe not
[17:53] <jcollado> s/find/fine/
[17:54] <jcollado> What do you think about having separate branches for tests depending on the distro?
[17:54] <ara> tests and library, isn't it?
[17:54] <ara> because they are very much interrelated
[17:54] <jcollado> ara: Correct. Test suite classes also. I missed that, thanks
[17:55] <jtatum> that might make sense... another possibility might be having a config document. something like default.xml (runs a couple of tests), karmic.xml (runs tests that work in karmic), etc.
[17:56] <jtatum> i know this is a funny time to suggest that, on the day eeejay's skip test cases branch got merged :)
[17:56] <ara> I would prefer jcollado's solution
[17:56] <cgregan> what percentage of cases only work on one version of Ubuntu?
[17:57] <ara> jtatum, problem with having that, is that the library might change because of a change in a string
[17:57] <mikefletcher> jtatum: I agree with Ara.  In the case of my gnome-screenshot the tests are different between Karmic and Jaunty.  You would want them to be in both releases.
[17:58] <jcollado> cgregan: I'd say around 25%, but that's just my own impression
[17:59] <ara> I have the QA meeting now. I have to leave you guys. I will read the backlog and send the notes to the list
[17:59] <jcollado> The problem is that when a test case is updated to the latest version, it's probably broken for the previous one
[18:00] <jtatum> right. but the obvious problem with branches is going to be merging and managing changes across 2-3 branches of tests :)
[18:00] <cr3> davmor2: thanks for pointing me to the bug report, the problem seemed to be package dependency or somesuch related, so I looked at the report.html file on cdimage.u.c but found no errors
[18:01] <davmor2> cr3: fader_ will tell you all about it
[18:02] <jcollado> jtatum: I agree
[18:02] <jcollado> jtatum: Your solution doesn't look bad, but the problem is that it doesn't provide the same test case for multiple distros simultaneously
[18:03] <jcollado> jtatum: Maybe we've got to discuss this further using the maillist
[18:03] <jtatum> no. it's not a great solution at all. in the case of mikefletcher's tests, there would have to be two test cases and two methods in gnome.py, one for jaunty and one for karmic. very ugly. branches are clearly better there.
[18:04] <jtatum> but in the case where pidgin shipped in all these versions of ubuntu and then no longer ships in karmic, a conf file is clearly better.
[18:04] <jtatum> i dunno. branches are probably the winner I guess :)
[18:05] <jcollado> jtatum: For that case a check to make sure that the application is installed would be enough as long as nothing else has to be changed
[18:06] <jtatum> jcollado: true indeed.
[18:07] <jcollado> jtatum: So is it ok to go for the multiple branches solution?
[18:08] <jtatum> jcollado: that seems to be the right answer, and there definitely seems to be consensus. I trust you guys to work out the specifics.
[18:08] <mikefletcher> +1 from me
[18:09] <jcollado> One think to note is that not all branches need the same level of support. Just the last one is supposed to get the new additions while the old ones could be updated just on improvements for test cases that are already included
[18:09] <jcollado> s/think/thing/
[18:10] <jcollado> This way, maintenance shouldn't be really hard
[18:10] <jtatum> jcollado: that makes a lot of sense. "trunk" for karmic until lucid becomes the target for new development, etc
[18:11] <jcollado> Great. I think that the meeting is finished then. Any other think you'd like to comment?
[18:13] <jcollado> Bye
[18:14] <mikefletcher> nope, thanks!
[18:14] <jtatum> good meeting, high fives all around and such

Testing/Automation/Meetings/20090923 (last edited 2009-09-24 07:23:50 by 226)