Ubuntu Wiki

   1 [17:00]  * ogasawara fades in
   2 [17:00] <dholbach> next up is Leann Ogasawara who will talk to us about "Kernel patches"!
   3 [17:00] <ogasawara> Hi Everyone!  Welcome to the Kernel Patches Session.
   4 [17:00] <ogasawara> My name is Leann Ogasawara and I'm a member of the Ubuntu Kernel Team.
   5 [17:00] <ogasawara> If you have any questions, please post them to #ubuntu-classroom-chat prefixed with QUESTION:
   6 [17:01] <ogasawara> One of the biggest issues we face as a team is the large volume of bug reports we receive against the kernel.
   7 [17:01] <ogasawara> One possible way to lend a helping hand is submitting patches to fix these bugs.
   8 [17:01] <ogasawara> Submitting a kernel patch might seem like a daunting task, but hopefully after today's session it won't seem so intimidating.
   9 [17:01] <ogasawara> There are five main items I'd like to cover in today's talk:
  10 [17:01] <ogasawara> 1) Choosing bugs to patch
  11 [17:01] <ogasawara> 2) Patch submission process
  12 [17:01] <ogasawara> 3) Walk through an actual example
  13 [17:01] <ogasawara> 4) Reasons for not applying a patch
  14 [17:01] <ogasawara> 5) Getting Help
  15 [17:02] <ogasawara> I should mention that the Ubuntu kernel team's ideal scenario is to have little to no divergence from the upstream linux kernel.
  16 [17:02] <ogasawara> If a kernel patch will benefit Ubuntu users, it'll likely benefit all.
  17 [17:02] <ogasawara> This is why it is important to make sure patches get submitted upstream.
  18 [17:02] <ogasawara> Also, keeping the delta we have between our kernel and the upstream kernel as small as we can makes it much much easier to maintain the kernel from our end.
  19 [17:03] <ogasawara> As such, the Ubuntu kernel team's preferred policy is for all patches to be submitted and accepted into the upstream kernel before pulling them into the Ubuntu kernel.
  20 [17:03] <ogasawara> So how does this happen?
  21 [17:03] <ogasawara> Let's begin with our first talking point.
  22 [17:03] <ogasawara> == Choosing bugs to patch ==
  23 [17:03] <ogasawara> Ideally we want to isolate good bug candidates that could easily be resolved if a patch were applied.
  24 [17:03] <ogasawara> A good approach might be to tackle the low hanging fruit first, ie bugs with patches already attached.
  25 [17:04] <ogasawara> Launchpad has the ability to search for kernel bugs with patches attached.
  26 [17:04] <ogasawara> https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bugs?field.has_patch=on
  27 [17:04] <ogasawara> As I had mentioned previously, due to the large volume of bugs, there are times when bugs are filed with patches attached to fix an issue but they unfortunately get lost in the noise.
  28 [17:04] <ogasawara> If we have additional volunteers trying to keep and eye on these types of quick fixes, it would be much appreciated.
  29 [17:05] <ogasawara> Even just getting these types of bugs into a triaged state would be a great help.
  30 [17:05] <ogasawara> If anyone is interested in getting involved, ping me after the session and I'be be happy to help get you started.
  31 [17:05] <ogasawara> Many times we see a reporter will open a bug and mention the issue can be fixed with a patch that exists upstream alread.
  32 [17:05] <ogasawara> Since the patch is already upstream, it's a simple process for a kernel developer to cherry pick the fix back into the Ubuntu kernel.
  33 [17:06] <ogasawara> That is one example of how patches get pulled from the upstream kernel into the Ubuntu kernel.
  34 [17:06] <ogasawara> However, not all bugs with patches can be resolved so easily.
  35 [17:06] <ogasawara> Sometimes a bug will have a patch attached but the patch itself has not been submitted/applied upstream nor to the Ubuntu kernel.
  36 [17:06] <ogasawara> These are the types of patches we want to focus on getting merged upstream and back into the Ubuntu kernel, assuming they are a valid fix.
  37 [17:07] <ogasawara> It's helpful to inquire with the patch author about the nature of the patch.
  38 [17:07] <ogasawara> Is the patch just a workaround for the issue but does not address the root cause?
  39 [17:07] <ogasawara> Was the patch only meant for debugging purposes to isolate the issue?
  40 [17:07] <ogasawara> We have to keep in mind that not all patches will be the proper fix.
  41 [17:07] <ogasawara> If the patch is a band aid and not a proper fix, it shouldn't be applied.
  42 [17:08] <ogasawara> On the other hand, if the patch author just needs assistance with submitting the patch, hopefully the information I talk about next will help.
  43 [17:08] <ogasawara> If anyone ever has a situation where they have a question regarding a patch they've seen or personally written, please don't hesitate to contact the kernel team.
  44 [17:08] <ogasawara> We're usually in the #ubuntu-kernel IRC channel on FreeNode and obviously also closely watch the Ubuntu kernel team mailing list.
  45 [17:08] <ogasawara> I'd like to take a second here to pause and field any questions before we move onto discussing the patch submission process.
  46 [17:09] <ogasawara> If there are no questions, we'll just move right along . . .
  47 [17:09] <ogasawara> Navaneeth> QUESTION: Patches applied on ubuntu kernel are sent back to upstream?
  48 [17:09] <ogasawara> Navaneeth: indeed we actually strive to push the patch upstream first, before applying to Ubuntu
  49 [17:10] <ogasawara> Navaneeth: that might not always be the case so after each release, we review our delta from upstream and work to get the non-ubuntu specific patches pushed up
  50 [17:11] <ogasawara> == Patch Submission Process ==
  51 [17:11] <ogasawara> First we need to confirm the patch does indeed resolve the issue.
  52 [17:11] <ogasawara> If you yourself can confirm the issue is resolved with the patch, that's perfect.
  53 [17:11] <ogasawara> Otherwise, we need to work with whomever has reported the issue and have them confirm the patch fixes the problem.
  54 [17:12] <ogasawara> Once we have confirmation the fix is valid, we're ready to send the patch upstream.
  55 [17:12] <ogasawara> If the bug the patch fixes happens to be reported in Launchpad, for completeness sake, the patch can be uploaded as an attachment to the bug report.
  56 [17:12] <ogasawara> When uploading the patch, be sure to select the "This attachment is a patch" option.
  57 [17:12] <ogasawara> Now, lets go through preparing the patch to submit upstream.
  58 [17:12] <ogasawara> Upstream has this process clearly documented at the following link.
  59 [17:12] <ogasawara> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/SubmittingPatches
  60 [17:13] <ogasawara> I'll highlight some of the more important bits below . . .
  61 [17:13] <ogasawara> 1. Know who/where to submit the patch.
  62 [17:13] <ogasawara>    This can be done by examining the MAINTAINERS file in the linux kernel tree
  63 [17:13] <ogasawara>    http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=MAINTAINERS
  64 [17:13] <ogasawara> 2. Be sure to create the patch against the upstream kernel source, not the Ubuntu kernel source.
  65 [17:13] <ogasawara>    http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git
  66 [17:14] <ogasawara>    Note that some subsystems maintain their own git trees.
  67 [17:14] <ogasawara>    If this is the case, create the patch against the subsystem's tree.
  68 [17:14] <ogasawara>    The location of the subsystem's tree can also be found in the MAINTAINERS file
  69 [17:14] <ogasawara> 3. Clearly describe what the patch is fixing.
  70 [17:14] <ogasawara>    Also be sure to break large patches up into small logical pieces.
  71 [17:15] <ogasawara>    A good rule of thumb is each patch should fix one specific issue.
  72 [17:15] <ogasawara> 4. Be sure to note the version of the kernel the patch applies to.
  73 [17:15] <ogasawara>    When submitting a patch upstream, the patch should apply cleanly to the latest version of the upstream kernel available.
  74 [17:15] <ogasawara> 5. Make sure the patch adheres to the kernel's coding style.
  75 [17:15] <ogasawara>    http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/CodingStyle
  76 [17:15] <ogasawara>   If you run scripts/checkpatch.pl, this should quickly detect any coding style violations.
  77 [17:16] <ogasawara> 6. Make sure to maintain full provenance of the patch by signing off on the patch.
  78 [17:16] <ogasawara>    Signing off on the patch allows others to see the full history of who's written/reviewed the patch.
  79 [17:16] <ogasawara>    Signing off is done by simply adding the "Signed-off-by:" line to your patch.
  80 [17:16] <ogasawara> 7. Send the patch as plain text in the email.
  81 [17:17] <ogasawara>    Do not attach the patch.
  82 [17:17] <ogasawara>    It's quicker to review a patch that's already present in the body of the email rather than having to download and open an attachment.
  83 [17:17] <ogasawara>    Your email client should provide a way to inline the text directly from the patch file into the body of the email.
  84 [17:17] <ogasawara>    Additionally, plain text allows others to respond and annotate their feedback directly in the context of the patch.
  85 [17:17] <ogasawara>    Be careful if you just cut and paste as this might have negative affects like switching tabs to spaces and therefore violating the kernel coding style.
  86 [17:18] <ogasawara>    Also, be careful that your email client doesn't mangle the patch by line wrapping the patch.
  87 [17:18] <ogasawara> 8. Prefix the subject of the email with "[PATCH]"
  88 [17:18] <ogasawara>    This allows people to immediately know this is a patch being submitted.
  89 [17:18] <ogasawara> 9. Don't give up.  Be persistent :)
  90 [17:18] <ogasawara>   If the patch is not immediately applied there are usually reasons.
  91 [17:19] <ogasawara>   Take suggestions constructively and fix up the patch if necessary.
  92 [17:19] <ogasawara>   Patches might go through a few rounds of reviews and iterations before being accepted.
  93 [17:19] <ogasawara>   If there is no response or feedback after submitting a patch, don't be afraid to submit it again.
  94 [17:19] <ogasawara> Eventually the patch should hopefully receive the required Ack's from upstream and get applied.
  95 [17:20] <ogasawara> Once the patch is applied upstream it will automatically make it's way into the Ubuntu kernel upon the next rebase.
  96 [17:20] <ogasawara> If the patch should be considered for a Stable Release Update to a previous kernel release, the patch must come via the upstream stable tree.
  97 [17:20] <ogasawara> In order for a patch to be accepted into the upstream stable tree, the patch must meet certain criteria.
  98 [17:20] <ogasawara> The patch should be relatively small in size (less than 100 lines of code).
  99 [17:20] <ogasawara> The patch must fix a real issue, no hypothetical use case fixes.
 100 [17:21] <ogasawara> The patch must already exist in Linus' tree upstream.
 101 [17:21] <ogasawara> For a full list of criteria, please refer to the following:
 102 [17:21] <ogasawara> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/stable_kernel_rules.txt
 103 [17:21] <ogasawara> Submitting a patch to the upstream stable tree is as easy as sending the patch in an email to stable at kernel.org
 104 [17:21] <ogasawara> Alternatively, when originally submitting a patch upstream, one can add a "CC:" line under their "Signed-off-by:" line.
 105 [17:22] <ogasawara> For a more detailed explanation of sending a patch to upstream stable, ssee the stable_kernel_rules.txt file referred to above.
 106 [17:22] <ogasawara> Once a patch has been submitted upstream, I usually like to add a reference to the upstream thread from the bug report.
 107 [17:22] <ogasawara> This allows anyone wanting to track the current status of the patch to read and follow along with the thread upstream.
 108 [17:22] <ogasawara> Again, if any patch author needs helps submitting a patch upstream, don't hesitate to ask those on the kernel team for help.
 109 [17:23] <ogasawara> I'm going to pause once more to field any questions.
 110 [17:23] <ogasawara> mirsal> QUESTION: What about using git to produce patches ?
 111 [17:23] <ogasawara> mirsal: of course, that's an excellent way to generate your patch
 112 [17:23] <ogasawara> mirsal: for those not familiar with git, the command it 'git format-patch'
 113 [17:24] <ogasawara> Navaneeth> QUESTION: Will this session covers debugging kernel source code and fixing a bug? If not, is there any helpful materials available online?
 114 [17:24] <ogasawara> Navaneeth: I'm going to walk through and example of a bug that was fixed and the patch pushed upstream
 115 [17:25] <ogasawara> Navaneeth: it likely won't go as in depth into the debugging process as you'd like, I'll point you to helpful materials after the session
 116 [17:25] <ogasawara> vishalrao_vbox> QUESTION: Do you @ubuntu have a hardware testing lab of sorts? For example can you test SSD support as you can see LP #502219 ?
 117 [17:25] <ogasawara> vishalrao_vbox: we do have access to hardware we maintain in a lab
 118 [17:25] <ogasawara> vishalrao_vbox: I do not know off the top of my head about the hw in the bug you note, but I'll take a look after the session
 119 [17:26] <ogasawara> any other questions?
 120 [17:26] <ogasawara> ok, we'll move along
 121 [17:27] <ogasawara> == Example of Submitting a Patch Upstream ==
 122 [17:27] <ogasawara> I thought it would be a good idea to walk through an actual bug whose patch was sent upstream.
 123 [17:27] <ogasawara> Lets take a look at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/366682
 124 [17:27] <ogasawara> This bug was about a system hanging on reboot.
 125 [17:27] <ogasawara> This particular bug was fixed by creating a patch to add a reboot quirk for this particular system.
 126 [17:28] <ogasawara> This patch would be beneficial not only for Ubuntu users but all users who had this hardware.
 127 [17:28] <ogasawara> As such, this was a good patch candidate to send upstream.
 128 [17:28] <ogasawara> From reading the report, you can see the original bug reporter was able to confirm the patch fixed the issue.
 129 [17:28] <ogasawara> I, the submitter, carefully read the upstream SubmittingPatches document and made sure to adhere to the items outlined in that document.
 130 [17:29] <ogasawara> Let's look at the actual email I posted:
 131 [17:29] <ogasawara> http://lists-archives.org/linux-kernel/27263461-x86-asus-p4s800-reboot-quirk.html
 132 [17:29] <ogasawara> 1) I tried to make sure I had a clear and well written description.
 133 [17:29] <ogasawara>    I made sure to clearly state what the issue was and what it is the patch was fixing.
 134 [17:29] <ogasawara>    I provided system information to help identify hardware that this patch would impact.
 135 [17:29] <ogasawara>    I made sure to note that the bug reporter had tested and confirmed the patch resolves the issue.
 136 [17:30] <ogasawara>    Additionally, I explained how and why it is I crafted the patch the way I did.
 137 [17:30] <ogasawara> 2) I made sure to follow the upstream kernel coding style
 138 [17:30] <ogasawara> 3) As you can see in the link, I inlined the patch within the body of the email
 139 [17:30] <ogasawara> 4) "[PATCH]" was prefixed in the subject line of the email
 140 [17:31] <ogasawara> 5) I examined the MAINTAINERS files to see that I should send the patch to the three subsystem maintainers (Thomas Gleixner, Ingo Molnar, H. Peter Anvin) and the x86 mailing list.  I also CC'd LKML for good measure.
 141 [17:31] <ogasawara> 6) I also saw from the MAINTAINERS file that they had their own linux-2.6-x86.git tree so I made sure my patch cleanly applied to the tip of this tree.
 142 [17:31] <ogasawara> 7) As a final task, I made sure I signed off on the patch by adding my "Signed-off-by:" line.
 143 [17:32] <ogasawara> The one thing that I failed to do was mention the version of the kernel this should be applied to.
 144 [17:32] <ogasawara> :/
 145 [17:32] <ogasawara> Thankfully the patch was accepted and applied even with this slight oversight.
 146 [17:32] <ogasawara> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4832ddda2ec4df96ea1eed334ae2dbd65fc1f541
 147 [17:32] <ogasawara> ^^ there's the patch in it's upstream form
 148 [17:32] <ogasawara> Additionally the maintainer CC'd upstream stable when applying this patch.
 149 [17:33] <ogasawara> As a result we have already had the patch come back to Karmic via the 2.6.31.9 upstream stable kernel.
 150 [17:33] <ogasawara> This is the typical scenario of how a patch goes upstream first and then makes it's way back into the Ubuntu kernel.
 151 [17:33] <ogasawara> There are obviously exceptions that are sometimes made.
 152 [17:33] <ogasawara> If a patch is for example a Critical bug fix, the Ubuntu kernel team may opt to temporarily carry the patch while it's going through the upstream submission process.
 153 [17:34] <ogasawara> The patch must still adhere to the usual patch submission policy and procedures.
 154 [17:34] <ogasawara> There are also cases where the patch is Ubuntu specific and therefore does not make sense being submitted upstream.
 155 [17:34] <ogasawara> For example, https://bugs.launchpad.net/ubuntu/+source/linux/+bug/393517
 156 [17:34] <ogasawara> This was an Ubuntu specific issue where we failed to completely remove a driver.
 157 [17:34] <ogasawara> If the patch is an Ubuntu specific patch, just send it to the Ubuntu kernel team mailing list.
 158 [17:35] <ogasawara> Additionally opening a bug in Launchpad to track the issue would also be good (and required when we go to fix the issue)
 159 [17:35] <ogasawara> Let's quickly pause again here for questions.
 160 [17:35] <ogasawara> Otherwise I'll keep going
 161 [17:36] <ogasawara> OK, lets keep moving
 162 [17:36] <ogasawara> == Reasons for not applying a patch ==
 163 [17:36] <ogasawara> If it isn't good enough for upstream it likely isn't good enough for us.
 164 [17:36] <ogasawara> That's not 100% true, but the basic idea is that if the patch is not in upstream, that generally means the patch is not likely ready for mass consumption.
 165 [17:37] <ogasawara> Ubuntu's kernel is not a testing ground for experimental features.  We need stability.
 166 [17:37] <ogasawara> As I mentioned earlier, maintaining patches in our tree is not an easy task if we have a large delta of divergence between our tree and the upstream tree.
 167 [17:37] <ogasawara> If we have a small delta to rebase, it makes maintaining the kernel much simpler.
 168 [17:37] <ogasawara> Also, if the code has deviated from what's upstream, it's hard to get any additional help from the upstream developers to debug any issues that may arise.
 169 [17:38] <ogasawara> There could also be security repercussions by carrying custom patches that are not upstream.
 170 [17:38] <ogasawara> The patch could contain potential security vulnerabilities, which we would likely never hear about since the code isn't part of the stock vanilla kernel and routinely being examined.
 171 [17:38] <ogasawara> Patching stock code could also introduce bugs.  Adding bugs is bad.
 172 [17:39] <ogasawara> It also might be that the patch is not against a new enough kernel.
 173 [17:39] <ogasawara> The issue and fix may not be relevant against the actively developed kernel.
 174 [17:39] <ogasawara> We also have to be careful about patches that may have a broad reaching impact.
 175 [17:39] <ogasawara> Even though a patch may fix an issue for one person, it may regress 10 others.
 176 [17:40] <ogasawara> Finally, the simplest explanation for why a patch might not be applied is that the patch might not be a legitimate patch which should be carried.
 177 [17:40] <ogasawara> Like I mentioned earlier, band aid's and hacky workarounds will not be applied wholesale.
 178 [17:40] <ogasawara> So what happens now if a patch was not accepted???
 179 [17:40] <ogasawara> There's usually a good reason why a patch wasn't accepted and it's usually explained in a response to the patch submission email.
 180 [17:41] <ogasawara> If there are suggestions for improvements that need to be completed before a patch will be accepted, make the improvements and resubmit.
 181 [17:41] <ogasawara> Most patches go through this type of review process and will need to be modified and re-submitted a few times before they are accepted.
 182 [17:41] <ogasawara> As I mentioned before, don't give up.
 183 [17:42] <ogasawara> In the end, there will however still be patches that never make it upstream nor into Ubuntu.
 184 [17:42] <ogasawara> An alternative if someone so chooses would be to roll their own kernel with the patch applied.
 185 [17:42] <ogasawara> Anyone should be able to do this by following the kernel compile guide
 186 [17:42] <ogasawara> https://help.ubuntu.com/community/Kernel/Compile
 187 [17:43] <ogasawara> Note that building and booting your own custom kernel is done purely at the users own risk
 188 [17:43] <ogasawara> Next week we're looking to clean up this Kernel/Compile guide to be more user friendly and easier to follow.
 189 [17:44] <ogasawara> I was going to pause once more in case there are any additional questions, otherwise I'll start to wrap it up
 190 [17:44] <ogasawara> vishalrao_vbox> QUESTION: after you submit a patch, how do you get notified whether it was accepted or rejected?
 191 [17:44] <ogasawara> vishalrao_vbox: typically you've receive an email from the maintainer that the patch has been applied
 192 [17:45] <ogasawara> vishalrao_vbox: otherwise they'll likely send feedback on what needs to be fixed up with the patch
 193 [17:45] <ogasawara> Navaneeth> QUESTION: Do you add new features to Ubunutu kernel without upstream having it? If yes, can you list out a feature like that?
 194 [17:45] <ogasawara> Navaneeth: we typically try to refrain from doing such a thing
 195 [17:46] <ogasawara> Navaneeth: again, it's that whole pain point of maintaining something out of tree
 196 [17:46] <ogasawara> anything else?
 197 [17:46] <ogasawara> vishalrao_vbox> QUESTION: what is this "SAUCE" we see mentioned in some ubuntu kernel changelogs?
 198 [17:46] <ogasawara> vishalrao_vbox: those are the Ubuntu specific patches
 199 [17:47] <ogasawara> vishalrao_vbox: those ones will likely never go back upstream as they don't make sense to go back upstream
 200 [17:47] <ogasawara> vishalrao_vbox> ogasawara: S.A.U.C.E. stands for? (acronym?)
 201 [17:48] <ogasawara> heh, I don't think it's actually an acronym in our usage of it, if it is I don't know what it stands for :)
 202 [17:48] <ogasawara> my understanding it implies it our "special sauce" patches
 203 [17:48] <ogasawara> anything else?
 204 [17:49] <ogasawara> vishalrao_vbox> yes, i figured, i was wondering if its all uppercase for a reason :)
 205 [17:49] <ogasawara> it's uppercase so it's easy for us to spot and search against in the logs
 206 [17:49] <ogasawara> we try to examine all our sauce patches each release and figure out why we have them and if they are still needed
 207 [17:50] <ogasawara> ok, lets move on to the last item
 208 [17:50] <ogasawara> == Getting Help ==
 209 [17:50] <ogasawara> I'd like to remind anyone that if you need assistance or have any questions regarding a patch, need a patch to be reviewed, or need help getting the patch submitted upstream, the Ubuntu kernel team would be happy to help.
 210 [17:50] <ogasawara> Please contact us through the kernel team mailing list or in the #ubuntu-kernel IRC channel on FreeNode.
 211 [17:51] <ogasawara> If you need to know the kernel team mailing list address, refer to our wiki.ubuntu.com/KernelTeam
 212 [17:51] <ogasawara> Also, most of what we covered today I'd previously documented at https://wiki.ubuntu.com/KernelTeam/KernelPatches
 213 [17:51] <ogasawara> I've also started graphing some trends for bugs with patches attached:
 214 [17:52] <ogasawara> http://qa.ubuntu.com/reports/ogasawara/csv-stats/bugs-with-patches/linux/
 215 [17:52] <ogasawara> It would be great to see if we can drive the number of bugs with patches attached down to 0.
 216 [17:52] <ogasawara> We've already made this a focus of previous kernel team bug days, but any additional help is appreciated.
 217 [17:52] <ogasawara> And one last shameless plea. . . if anyone is interested in getting involved but just doesn't know where to start, let me know!
 218 [17:53] <ogasawara> I'll be glad to help get you started.
 219 [17:53] <ogasawara> Anyways, thanks again to all for listening.  I'll finish to fielding the remaining questions.
 220 [17:53] <ogasawara> as time permits :)
 221 [17:53] <ogasawara> vocx> QUESTION: what percentage of bug reports filed against the kernel are invalid? I'm under the impression that a lot of new users incorrectly file bug reports against the kernel, which contributes to much "noise".
 222 [17:54] <ogasawara> vocx: it is true that the kernel does get a large amount of bugs that are not valid against the kernel, but they are still valid in a sense that it's an issue against a different package
 223 [17:54] <ogasawara> vocx: I can't quote an exact number unfortunately
 224 [17:55] <ogasawara> vocx: but we do appreciate the community triage help on making sure the bug finds it's way to the proper package
 225 [17:55] <ogasawara> vocx> QUESTION: what percentage of patches are rejected?
 226 [17:55] <ogasawara> vocx: I'd say a very small percentage of patches are outright rejected
 227 [17:55] <ogasawara> vocx: probably less than 5%?
 228 [17:56] <ogasawara> vocx: If an issue is legitimate it can stay open
 229 [17:56] <ogasawara> vocx: patches are always welcome :)
 230 [17:56] <ogasawara> vocx> QUESTION: how much experience do you need to be able to submit a patch to the Linux-freaking-Kernel?
 231 [17:57] <ogasawara> vocx: you will need to have a general understanding of the kernel to know that the patch is correct
 232 [17:57] <ogasawara> vocx: but it's not rocket science :)
 233 [17:57] <ogasawara> <ecm123> QUESTION: I've heard the Nouveau driver was going to be included in in lucid's kernel, however lucid will not use the 2.6.33 kernel in which this driver is included by default, does that mean the ubuntu kernel team will maintain the driver all by itself in lucid?
 234 [17:58] <ogasawara> ecm123: yes there are talks/works to have this backported and available
 235 [17:59] <ogasawara> anything else?
 236 [17:59] <ogasawara> 1min . . .
 237 [17:59] <ogasawara> Navaneeth> QUESTION: As promised, can we have the online materials for debugging kernel source?
 238 [17:59] <ogasawara> of course, give me a few minutes to gather the links and I'll post it to ubuntu-classroom-chat so not to run over into the next session
 239 [18:00] <ogasawara> ok, that's it for me
 240 [18:00] <ogasawara> thanks everyone!