Ubuntu Wiki

Agenda

• Review ACTION points from previous meeting
  • mathiaz to fix the "server fixed bugs" broken script
  • ttx to confirm with jib lucid SRU resource allocation

• Release status (ttx)

  • Known Issues (release notes)
  • Targets for early SRUs
  • Remaining work items
• UEC release status (kirkland)
• EC2 release status (smoser)

• Weekly Updates & Questions for the QA Team (hggdh)

  • Release status

• Weekly Updates & Questions for the Kernel Team (jjohansen)

  • Release status
  • Ext4 performance regressions addressed?

• Weekly SRU review: https://wiki.ubuntu.com/ServerTeam/KnowledgeBase#SRU%20weekly%20review (mathiaz)

• Final call for blueprints for UDS Maverick
• Open Discussion
• Announce next meeting date and time
  • New proposed meeting time starting the Maverick cycle: Tuesday 6 pm UTC

Minutes

Actions

1. hggdh, zul to get their release note/early SRU bugs known to ttx
2. ALL to mark DONE or POSTPONED the remaining work items for 10.04 by Thu 29 April EOD
3. hggdh to outline testing bottlenecks for UEC testing UDS sessions

Key Points

Key resolutions or discussions are highlighted in the full log below by a leading -->

Agree on next meeting date and time

Next meeting will be on Wednesday, May 4th at 18:00 UTC in #ubuntu-meeting.

Log

[15:00] <kirkland> o/
[15:01] <jiboumans> morning folks
[15:01] <jiboumans> #startmeeting
[15:01] <MootBot> Meeting started at 09:01. The chair is jiboumans.
[15:01] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[15:01] <jiboumans> [TOPIC] Review ACTION points from previous meeting
[15:01] <MootBot> New Topic:  Review ACTION points from previous meeting
[15:01] <jiboumans> mathiaz to fix the "server fixed bugs" broken script
[15:01] <mathiaz> jiboumans: not done
[15:02] <jiboumans> mathiaz: when can we expect it to be tackled?
[15:02] <mathiaz> jiboumans: this week - now that release is out of the door
[15:02] <jiboumans> mathiaz: ack
[15:02] <jiboumans> ttx to confirm with jib lucid SRU resource allocation
[15:02] <ttx> jiboumans: I mentioned it to you
[15:02] <mathiaz> jiboumans: I should have some more time to look around these lower important todos
[15:02] <ttx> jiboumans: not sure that confirms it yet
[15:03] <jiboumans> ttx: i believe we have a UDS session on this, no?
[15:03] <ttx> jiboumans: yes
[15:03] <jiboumans> mathiaz: it happens, improvement point for next time :)
[15:03] <ttx> "Lucid SRU process review"
[15:03] <jiboumans> ttx: excellent, consider it done then
[15:03] <jiboumans> now, this meeting will focus on the releaes and UDS instead of the usual topics, so bear with us please
[15:04] <jiboumans> [TOPIC] Release status (ttx)
--> [15:04] <MootBot> New Topic:  Release status (ttx)
[15:04] <jiboumans> how's the war room looking ttx?
[15:04] <ttx> yay
[15:04] <ttx> DefCon 3 so far
--> [15:04] <ttx> No respin expected
[15:04] <mathiaz> ttx: what does that mean?
[15:04] <mathiaz> ttx: level 3?
[15:04] <ttx> nobody running in circles
[15:04] <nealmcb> :)
[15:04] <jiboumans> mathiaz: you're woefully underprepared for an invasion of your borders sir
[15:05] <zul> drinking yet?
[15:05] <ttx> DefCon 2 would be "pilots scramble to their aircrafs
[15:05] <mathiaz> jiboumans: totally - where I live there isn'
[15:05] <ttx> and DefCon 1 would be permission to engage
[15:05] <mathiaz> jiboumans: totally - where I live there isn't any point in getting prepare for an invasion
[15:05] <zul> release lucid....drunk fix later..
[15:05] <ttx> the absinthe bottle is still full
[15:05]  * hggdh meanwhile drinks a marvelous Brazilan coffee
[15:05] <jiboumans> ttx: any known issues come up that will be release noted or SRU'd early?
[15:06]  * zul hands mathiaz a book about the war of 1812
[15:06] <Daviey> I understand it's tradition for the server boss to buy drinks at UDS for the entire ~ubuntu-server team.. so it's not all bad, eh jiboumans ?
[15:06] <ttx> yes
[15:06] <jiboumans> Daviey: i must have not gotten that memo
[15:06] <ttx> I updated https://wiki.ubuntu.com/ServerTeam/Meeting
[15:06] <mathiaz> jiboumans: you'll get - don't worry
[15:06] <ttx> ah
[15:06] <ttx> I mean https://wiki.ubuntu.com/ServerTeam/ReleaseStatus
[15:07] <jiboumans> ttx++ excellent
[15:07] <ttx> We have 4 things ion the release note list
[15:07] <ttx> 4 early targets for SRU
[15:07] <ttx> If you see anything missing, please shout
[15:07] <ttx> (now)
[15:07] <hggdh> ttx, may I propose bug 567592 also?
[15:07] <ubottu> Launchpad bug 567592 in plymouth "rm: cannot remove `/var/lib/urandom/random-seed': Read-only file system" [Undecided,New] https://launchpad.net/bugs/567592
[15:07] <jiboumans> for the logs:
[15:07] <jiboumans> 423252 (breaks NSS using LDAP+SSL on upgrades from Hardy)
[15:07] <jiboumans> 571057 (breaks OpenLDAP configs on upgrades from Lucid dev milestones)
[15:07] <jiboumans> 570805 (regression in Ubuntu Server installation times on ext4)
[15:07] <jiboumans> 557429 (potential breakage in a scenario with degraded RAID arrays being assembled)
[15:07] <zul> ttx: i know at least one other php one
[15:08] <ttx> hggdh: for release noting ?
[15:08] <mathiaz> bug 423252 <- that won't be SRUable in the short time
[15:08] <ubottu> Launchpad bug 423252 in sudo "NSS using LDAP+SSL breaks setuid applications like su and sudo" [High,Confirmed] https://launchpad.net/bugs/423252
[15:08] <hggdh> ttx: for SRU-ing
[15:08] <mathiaz> as there isn't a confirmed fix
[15:08] <hggdh> early
[15:08] <jiboumans> hang on, those were the release note bugs
[15:08] <ttx> mathiaz: right that's why it's on the "under investigation" list
--> [15:08] <jiboumans> these are the SRUable bugs:
--> [15:09] <jiboumans> eucalyptus: 567371,565101 (kirkland)
--> [15:09] <jiboumans> tomcat6: 563642 (ttx)
--> [15:09] <jiboumans> php5: 541439 (zul)
--> [15:09] <jiboumans> vmbuilder: 536942,386463,493020,497258,517067,535397,536940,539727,540446,524020 (soren)
[15:09] <zul> heh soren has some work to do
[15:09] <ttx> yes, volcano smoke prevented him to fix those in time
--> [15:09] <kirkland> jiboumans: i'm prepping a eucalyptus upload right now that addresses 566793, 567371, 565101
[15:09] <ttx> hggdh: ok, I'll look into it
[15:10] <ttx> kirkland: i'll add the missing bugreference, thanks for the pointer
[15:10] <jiboumans> [ACTION] hggdh, zul to get their release note/early SRU bugs known to ttx
[15:10] <MootBot> ACTION received:  hggdh, zul to get their release note/early SRU bugs known to ttx
[15:10] <jiboumans> ttx: any other news from the war room?
[15:10] <zul> ttx: what about the crypt lvm bug?
[15:10] <ttx> zul: the repeated prompt ?
[15:10] <zul> yeah shouldnt that be in the release notes as well?
[15:10] <ttx> zul: I don't think so
[15:11] <Daviey> Is soren going to be able to manage the *long* list of early SRU's?
[15:11] <ttx> it sucks but it's hardly blocking anything
[15:11] <zul> ttx: ack
[15:11] <ttx> Daviey: patches welcome, as usual
[15:11] <ttx> nothing else from the war room
--> [15:11] <jiboumans> [TOPIC] UEC release status (kirkland)
[15:11] <MootBot> New Topic:  UEC release status (kirkland)
[15:11] <jiboumans> thanks ttx
[15:11] <kirkland> jiboumans: check
--> [15:12] <kirkland> jiboumans: no release critical issues
[15:12] <ttx> just remember to mark DONE or POSTPONED the remaining work items for 10.04
[15:12] <ttx> by tomorrow eod :)
[15:12] <jiboumans> [ACTION] ALL to mark DONE or POSTPONED the remaining work items for 10.04 by Thu 29 April EOD
[15:12] <MootBot> ACTION received:  ALL to mark DONE or POSTPONED the remaining work items for 10.04 by Thu 29 April EOD
[15:13] <jiboumans> kirkland: the upload you're doing will be a zero day SRU then?
[15:13] <kirkland> jiboumans: i have uploaded now
=== yofel is now known as yofel_
[15:13] <kirkland> jiboumans: it will be in lucid-proposed as soon as the ubuntu-sru team accepts it
[15:13] <kirkland> jiboumans: normally, it's supposed to bake in -proposed for ~1 week
[15:13] <jiboumans> kirkland: right, but too late for any ISOs of course
[15:13] <jiboumans> understood
=== yofel_ is now known as yofel__
[15:14] <kirkland> jiboumans: yes, these won't be on the 10.04 iso (but will be on 10.04.1)
[15:14] <jiboumans> kirkland: does it address all the open bugs we'd hope to have fixed real soon around release, or do we have any open ones left?
=== yofel__ is now known as yofel
[15:14] <kirkland> jiboumans: we could ask for a rush processing of the 1 week bake, since we've been testing it for nearly a week already now
[15:14] <ttx> jiboumans: those are mostly bugs you encounter running over a period of time (not installer issues) so updating is ok
[15:14] <kirkland> jiboumans: these are mostly scalability one
[15:14] <kirkland> jiboumans: i suspect as soon as people actually start deploying 10.04 UEC, we'll get some new, hard ones
[15:15] <jiboumans> kirkland: i'm sure
[15:15] <kirkland> jiboumans: i think ttx had a few more he wanted looked at
[15:15] <ttx> yes, the securitygroup thing I haven't had time to reproduce
[15:15] <kirkland> jiboumans: and smoser still has some upstarty/plymouth-ish issues, ISTR
[15:15] <jiboumans> kirkland: for ec2, surely. we'll cover that next
[15:15] <ttx> yes our images are still pretty brittle when run over UEC
[15:16] <jiboumans> then let's cover that next
--> [15:16] <jiboumans> [TOPIC] EC2 release status (smoser)
[15:16] <smoser> not brittle
[15:16] <MootBot> New Topic:  EC2 release status (smoser)
[15:16] <jiboumans> thanks kirkland, tt
[15:16] <jiboumans> *ttx
--> [15:16] <smoser> in most cases things are fine. under high heavy load on UEC the metadata service doesn't come up for minutes.
[15:16] <smoser> the images don't wait long enough
[15:16] <smoser> so yes, maybe brittle
[15:17] <jiboumans> ttx: i trust that is noted somewhere?
[15:17] <ttx> I meant the "can't be contacted by ssh" thing
[15:17] <ttx> jiboumans: it's difficult to releasenote it since we have trouble to understand when it hits
[15:17] <smoser> hold on. getting bugs.
[15:17] <ttx> "UEC image may fail to boot on UEC" isn't a good releasenote text
[15:17] <smoser> bug 565018
[15:17] <ubottu> Launchpad bug 565018 in cloud-init "instance is not reachable via ssh" [High,Confirmed] https://launchpad.net/bugs/565018
[15:18] <smoser> bug 566792
[15:18] <ubottu> Launchpad bug 566792 in eucalyptus "UEC guests sometimes fail on consuming user data (metadata service isn't ready)" [High,Confirmed] https://launchpad.net/bugs/566792
[15:18] <ttx> the second one is under load, right
[15:18] <smoser> i hope to address 566792 sometime very soon after release, or at least part of it.
[15:19] <ttx> but the first one can be hit on a single try, if you're unlucky
[15:19] <smoser> the first bug (instance not reachable via ssh), i have no good explanations under what conditions it fails
[15:19] <smoser> it seems timing related.
[15:19] <hggdh> 566792 seems to happen more when CLC, Walrus, and CC are on differetn machines
[15:19] <smoser> Hmm..
[15:20] <smoser> i guess there should be a third bug. i will open one.
[15:20] <smoser> the 'meta data service comes up slowly' portion of bug 566792 is easily workaroundable in image
[15:20] <ubottu> Launchpad bug 566792 in eucalyptus "UEC guests sometimes fail on consuming user data (metadata service isn't ready)" [High,Confirmed] https://launchpad.net/bugs/566792
[15:20] <smoser> but your boots will just take 3 minutes
[15:21] <jiboumans> beats not booting at all
[15:21] <smoser> the "meta data service returned empty data an http status 200" is not so easily workaroundable.
[15:21] <smoser> that is much harder to reproduce.
[15:21] <smoser> opening bug now.
--> [15:21] <jiboumans> smoser: any issues with the images themselves?
[15:21] <smoser> well, no. :)
--> [15:22] <smoser> not after they're booted
[15:22] <jiboumans> smoser++ excellent
[15:22] <hggdh> er
[15:22] <ttx> jiboumans: they run relatively fine on EC2
[15:22] <hggdh> there is bug 567592, hitting plymouth, and may be germane here
[15:22] <ubottu> Launchpad bug 567592 in plymouth "rm: cannot remove `/var/lib/urandom/random-seed': Read-only file system" [Undecided,New] https://launchpad.net/bugs/567592
[15:22] <jiboumans> ttx: when you say 'relatively' i expect you can tell me when they don't?
[15:22] <smoser> very good on ec2.
[15:22] <ttx> We could have a generic "UEC image may fail to boot on UEC" release note pointing to the bugs still in progress
[15:23] <ttx> jiboumans: I meant "comparatively"
[15:23] <smoser> hggdh, how often do you see that bug ?
[15:23] <hggdh> smoser: every boot I have the console log I see it. I think this may play a role
[15:24] <smoser> "have the console log" ?
[15:24] <hggdh> not the cannot remove, but the ral cause -- plymouth barfs when trying to write to a pts
[15:24] <nealmcb>  
[15:24] <hggdh> smoser: the failed instances
[15:24]  * nealmcb oops
[15:24] <smoser> i dont think thats true.
[15:25] <smoser> you're saying every time you can't reach an instance it has that in it.
[15:25] <smoser> ?
[15:25] <hggdh> I am saying every UEC image boot has it -- plymouth dies with a SIGSEGV
[15:25] <hggdh> and this may cause problems for mountall, and cascade
[15:26] <smoser> correct.
[15:26] <smoser> but not every instance fails in that bug (read only filesystem)
[15:26] <hggdh> indeed, no. But the read-only is (as Collin found) a *possible* consequence, not the cause
[15:27] <smoser> ok.
[15:27] <smoser> i wasn't read-up-to-date on that
[15:27] <hggdh> this is why I would really like to have a new UEC image with Collin's fixes, to test it again
[15:27] <ttx> hggdh: we have a fix for that already ?
[15:28] <hggdh> cjwatson just proposed fixes for it, this morning
[15:29] <ttx> right, we expect to roll Lucid UEC images regularly, so we could have an updated image with the fix quite fast if that fixes it
[15:29] <ttx> there are a few other boot-related fixes in the queue
[15:30] <jiboumans> ok, thanks for that
[15:30] <jiboumans> moving on
[15:30] <jiboumans> [TOPIC] Weekly Updates & Questions for the QA Team (hggdh)
--> [15:30] <MootBot> New Topic:  Weekly Updates & Questions for the QA Team (hggdh)
--> [15:30] <jiboumans> my question: release status? :)
--> [15:30] <hggdh> jiboumans: seems as ready as possible, with the given exceptions (bugs) above
[15:31] <zul> jiboumans: not paniacing
[15:31] <jiboumans> hggdh: that's great news
[15:31]  * jiboumans attaches the head back to his chicken
[15:31] <smoser> bug 571271 opened for "simple timeout issue"
[15:31] <ubottu> Launchpad bug 571271 in cloud-init "uec images should wait longer for metadata service" [High,Triaged] https://launchpad.net/bugs/571271
[15:31] <hggdh> another point
[15:32] <hggdh> I am adding a blueprint (joint QA/Server) for UEC testing. Even with mathiaz scripts (and bless him!), it is extremely costly in time to run the test
[15:33] <hggdh> even more with the dpkg/fsync currently in place :-(
[15:33] <jiboumans> hggdh: engineering or machine time?
[15:33] <hggdh> both of them
[15:33] <hggdh> a re-setup usually takes some two hours, and each stress about another 2 hours
[15:34] <jiboumans> that's costly indeed
[15:34] <jiboumans> let's take some time at UDS to see how we can improve that timewise. if it takes extra hardware, we can probably arrange that too
[15:35] <hggdh> thank you. I will add the blueprint in a few
[15:35] <jiboumans> [ACTION] hggdh to outline testing bottlenecks for UEC testing UDS sessions
[15:35] <MootBot> ACTION received:  hggdh to outline testing bottlenecks for UEC testing UDS sessions
[15:35] <jiboumans> hggdh: any ohter updates from the QA team?
[15:35] <hggdh> no, life is good now ;-)
[15:35] <jiboumans> any questions for hggdh?
[15:36] <jiboumans> i'll take that as a no
[15:36] <jiboumans> [TOPIC] Weekly Updates & Questions for the Kernel Team (jjohansen)
--> [15:36] <MootBot> New Topic:  Weekly Updates & Questions for the Kernel Team (jjohansen)
[15:36] <jiboumans> hggdh: thank you
[15:36] <jjohansen> hi
[15:36] <jiboumans> hi there
--> [15:36] <jiboumans> my question is unchanged: release status? :)
=== Claudinux_ is now known as Claudinux
[15:37] <jjohansen> heh, well unfortunately not really changed since last week
--> [15:37] <jjohansen> the kernel has been frozen and only a couple critical changes have been applied
--> [15:38] <jjohansen> everything is being queued for SRU
[15:38] <jiboumans> any of those changes or SRU affecting server issues?
[15:38] <jjohansen> no
[15:38] <jiboumans> jjohansen: ok, thanks
[15:39] <jiboumans> i had one specific question about the ext4 performance degrade we saw in benchmarks
[15:39] <jjohansen> okay
[15:39] <jiboumans> this is a public one of those:   http://www.phoronix.com/scan.php?page=article&item=ubuntu_lts_perf&num=2
[15:39] <jiboumans> it's a few weeks old though; have we managed to do anything with/about that ?
[15:40] <jjohansen> hrmm, I actually haven't seen anything about that pass by but give me a sec to check with csurbi
[15:41] <jjohansen> if I recall correctly this particular test wasn't really hitting the disk but I wasn't the one looking into it
[15:42] <jjohansen> jiboumans: I will have to get back to you on specifics as csurbi isn't around atm
[15:42] <jiboumans> jjohansen: no problem. could you coordinate with ttx to see if we should release note anything?
[15:42] <jjohansen> will do
[15:42] <jiboumans> thank you
[15:43] <jiboumans> any other questions or updates for the kernel?
[15:43] <jjohansen> one other little thing, it looks like trying to go pv-ops for EC2 will be a target for M
[15:43] <zul> cool...good luck! ;)
[15:44] <jiboumans> jjohansen: nice to hear
[15:44] <jiboumans> alright, moving on
[15:44] <jiboumans> [TOPIC] Weekly SRU review: https://wiki.ubuntu.com/ServerTeam/KnowledgeBase#SRU%20weekly%20review (mathiaz)
--> [15:44] <MootBot> New Topic:  Weekly SRU review: https://wiki.ubuntu.com/ServerTeam/KnowledgeBase#SRU%20weekly%20review (mathiaz)
[15:44] <jiboumans> i expect very little, but best to check :)
--> [15:45] <mathiaz> no bugs nominated for any release
[15:45] <mathiaz> anything worth SRU in the last week?
--> [15:45] <mathiaz> and next week I'll add lucid to the list
--> [15:45] <mathiaz> so expect a longer SRU review as we go through the lucid nominations
[15:45] <zul> nope
[15:46] <mathiaz> it may require more than one meeting though
[15:46]  * jiboumans renames this the SRU meeting
[15:46] <jiboumans> mathiaz: understood and thanks
[15:46] <jiboumans> [TOPIC] Final call for blueprints for UDS Maverick
--> [15:46] <MootBot> New Topic:  Final call for blueprints for UDS Maverick
[15:47] <jiboumans> Starting monday, we'll do the planning for the sessions of UDS maverick
--> [15:47] <jiboumans> if you'd like to have a guaranteed session slot, please make sure your blueprint is created by this Friday end of business
--> [15:48] <jiboumans> Please also take a look here for currently proposed sessions at UDS: https://wiki.ubuntu.com/ServerTeam/MaverickIdeaPool
--> [15:48] <jiboumans> if you have any ideas, no matter how experimental, feel free to add those to the above wiki page as well
[15:49] <jiboumans> [TOPIC] Open Discussion
[15:49] <MootBot> New Topic:  Open Discussion
[15:49] <jiboumans> please, go ahead :)
[15:50] <jiboumans> alright, moving on
[15:50] <jiboumans> [TOPIC] New meeting time for the Maverick cycle
--> [15:50] <MootBot> New Topic:  New meeting time for the Maverick cycle
[15:51]  * hggdh really pays attention now
[15:51] <jiboumans> *drum roll*
[15:51] <ttx> let's all act surprised !
[15:51] <RoAkSoAx> lol
--> [15:51] <jiboumans> Given that release days/freezes are on Thursday, we'd like to move the meeting to Tuesday 6 pm UTC
[15:51] <hggdh> YES!
[15:51] <jiboumans> this aligns better with west coast attendance (hi jjohansen)
[15:52] <hggdh> (sorry, ttx)
[15:52] <ttx> 1800 UTC, you mean
[15:52] <jiboumans> and is one of the few slots left on the Tuesday meeting schedule
[15:52] <jjohansen> yes it does
[15:52] <jjohansen> :)
[15:52] <jiboumans> ttx: if it makes you feel better
[15:52] <jiboumans> all those opposed say 'nay'
[15:52] <ttx> that makes me feel more accurate :P
[15:52] <jiboumans> no objections, the new meeting time stands
[15:53] <jiboumans> \o/
[15:53]  * hggdh agrees with ttx
[15:53] <Daviey> (if it's changed, the date on the ubuntu fridge cal needs updating)
[15:53] <hggdh> oh, no early mornings...
[15:53] <jiboumans> [TOPIC] Announce next meeting date and time
[15:53] <MootBot> New Topic:  Announce next meeting date and time
[15:53] <jiboumans> kirkland, ttx and myself will be at a company meeting next week, and UDS is the week after
[15:53] <mathiaz> I can run the meeting next Tuesday
[15:53] <jiboumans> is there mileage in a meeting next week, or shall we postpone till after UDS?
[15:54] <jiboumans> mathiaz: thanks for the offer. worth it for you to go through SRUs etc?
[15:54] <ttx> hggdh: oh, meetings after dinner
[15:54] <mathiaz> jiboumans: yes
[15:54] <jiboumans> ok, then
[15:54] <soren> jiboumans: Will you? Even with the new meeting time?
[15:54] <mathiaz> even if it lasts just 10 minutes
[15:54] <jiboumans> soren: i will always. not sure what i will, but i will
[15:55] <jiboumans> mathiaz: excellent
--> [15:55] <jiboumans> next meeting time is Tuesday May 4th at 1800 UTC
[15:55] <soren> jiboumans: Be at the company meeting at the time of the meeting.
[15:55] <jiboumans> soren: technically, i'll be in a plane over the atlantic
[15:55] <soren> Maybe my intel is wrong :)
[15:55] <ttx> soren: your intel is wrong
[15:55] <jiboumans> thanks for highlighting that crucial distinction :)
[15:55] <jiboumans> alright all, thanks for attending
[15:55] <jiboumans> #endmeeting
[15:55] <MootBot> Meeting finished at 09:55.
[15:56] <nealmcb> sounds like mileage in that meeting to me :)
[15:56] <Daviey> jiboumans: New meeting time is long term?
[15:56] <mathiaz> Daviey: yes :)
[15:56] <Daviey> okay, thanks
[15:56]  * Daviey prods the firdge people
[15:56] <Daviey> fridge*
[15:57] <jiboumans> Daviey: thank you