Ubuntu Wiki

Dev Week -- Kernel Triaging and Debugging -- ogasawara -- Tue, Sep 1st, 2009

UTC

(01:01:02 PM) dholbach: next up is Leann Ogasawara who will help us dive into Kernel Triaging and Debugging!
(01:01:16 PM) ogasawara: Hi Everyone!  Welcome :)
(01:01:29 PM) ogasawara: My name is Leann Ogasawara and I help manage the Ubuntu Kernel Team's incoming and existing bugs against the kernel.
(01:01:30 PM) frandieguez: hi ogasawara
(01:01:45 PM) ogasawara: Having to deal with such a large volume of bugs is always a huge challenge for us.
(01:01:56 PM) dholbach: note: chat and questions please in #ubuntu-classroom-chat
(01:01:56 PM) ogasawara: I thought this session would be a good opportunity to share some best practices I've learned along the way for triaging kernel bugs as well as share some information for helping debug issues.
(01:02:14 PM) ogasawara: We're always looking for more involvement from the community to help triage kernel bugs so hopefully after today's session some of you might be interested in helping out.
(01:02:28 PM) ogasawara: Let's first start with what the role of a kernel bug triager is.
(01:02:41 PM) ogasawara: The goal for any kernel bug triager is to get a bug into a state such that a developer can immediately being working on a fix.
(01:02:53 PM) ogasawara: Remember, as a triager we are often the first point of contact for a bug reporter.
(01:03:04 PM) ogasawara: It's important that we help move a bug into a good working state as well as help educate the bug reporter to submit better bug reports in the future.
(01:03:21 PM) ogasawara: So how does that happen?
(01:03:27 PM) ogasawara: First, we help make sure Ubuntu kernel bugs are assigned to the Ubuntu linux kernel package.
(01:03:38 PM) ogasawara: http://bugs.launchpad.net/ubuntu/+source/linux
(01:03:44 PM) ogasawara: For example, if someone is experiencing a kernel oops or panic, then that's obviously a kernel bug.
(01:03:55 PM) ogasawara: If a bug reporter did not correctly file the bug against the linux kernel package, help reassign the bug and kindly remind them to report future kernel bugs against the linux kernel package.
(01:04:11 PM) ogasawara: Failing to do so may result in the bug getting overlooked.
(01:04:22 PM) ogasawara: It may be helpful to also point them at https://wiki.ubuntu.com/Bugs/FindRightPackage .
(01:04:34 PM) ogasawara: Next, we want to make sure a bug is really not a duplicate of another bug.
(01:04:43 PM) ogasawara: This is where we as kernel triagers need to be careful.
(01:04:54 PM) ogasawara: Kernel bugs are usually hardware specific.
(01:05:07 PM) ogasawara: Just because someone may be experiencing the same symptom of another bug reporter doesn't necessarily mean they have the same bug.
(01:05:20 PM) ogasawara: When in doubt, don't mark it as a duplicate and ask for a second opinion.
(01:05:33 PM) ogasawara: Additionally, if you see someone comment on a bug and they don't have the same hardware, ask them to open a new bug report and explain why.
(01:05:47 PM) ogasawara: This really helps avoid bugs becoming wildly out of control and impossible for a developer to follow, let alone fix.
(01:06:04 PM) ogasawara: Next, help make sure the title of the bug as well as the bug description is informative.
(01:06:15 PM) ogasawara: "Sounds is broken" or "Suspend fails" is not informative.
(01:06:26 PM) ogasawara: Like I mentioned above, kernel bugs are usually hardware specific.  It's always best to mention the affected hardware in the title as well as the bug description.
(01:06:46 PM) ogasawara: This will again help avoid bugs becoming a mess.
(01:06:55 PM) ogasawara: If you find you have the same hardware as a bug being reported, try to reproduce the bug yourself.
(01:07:08 PM) ogasawara: t's not unheard of for hardware to become faulty.
(01:07:21 PM) ogasawara: Being able to help confirm this is or is not the result of hardware going bad is important.
(01:07:30 PM) ogasawara: Also, be sure to document the steps to reproduce if possible.
(01:07:42 PM) ogasawara: Now I know the next part is sometimes a bit controversial, but it's also best if the issue has been confirmed against the latest kernel available.
(01:07:52 PM) lopo1 left the room.
(01:07:59 PM) ogasawara: I realize this is a touchy subject for some individuals and some reporters often object to always being asked to "test the latest".
(01:08:13 PM) ogasawara: However, when you are dealing with the kernel, keep in mind there are literally thousands of commits between each release.
(01:08:24 PM) ogasawara: Then consider that each commit touches more than just one line of code and you've now hit insanity trying to isolate one fix (if it even exists) for a single bug.
(01:08:51 PM) ogasawara: Finally, one of the most important aspects of traiging kernel bugs is making sure the appropriate log information is attached.
(01:09:07 PM) ogasawara: For the kernel this means dmesg output, lspci, kernel version info, etc.
(01:09:20 PM) ogasawara: I recommend that instead of asking bug reporters to attach these files individually, have them run apport-collect.
(01:09:33 PM) ogasawara: apport-collect will automatically gather and attach general linux kernel debug information for a bug.
(01:09:55 PM) ogasawara: For example, if we wanted kernel debug info attached to pretend bug 987654, the apport-collect command would look like:
(01:10:03 PM) ogasawara: apport-collect -p linux 987654
(01:10:22 PM) ogasawara: The nice part about this is we're only asking the bug reporter to run a single command.
(01:10:35 PM) ogasawara: There's less room for error having a bug reporter run one command versus having a bug reporter run multiple commands.
(01:10:48 PM) ogasawara: Even better than reporting a bug and running apport-collect on it, is to use ubuntu-bug to report the bug in the first place.
(01:11:01 PM) ogasawara: This will automatically file the bug against the linux kernel package and again automatically gather and attach kernel debug info to the new bug.
(01:11:15 PM) ogasawara: The command looks like the following:
(01:11:23 PM) ogasawara: ubuntu-bug -p linux
(01:11:36 PM) ogasawara: In the process of attempting to triage a bug, if you've asked a bug reporter to provide more information, be sure to set the bug's status to Incomplete.
(01:11:52 PM) ogasawara: Also be sure to subscribe yourself to a bug so that you are automatically notified when they have responded with the requested information.
(01:12:09 PM) ogasawara: Once the bug looks ready for a developer to begin working on it, set the status of the bug to Triaged and make sure the Importance is set.
(01:12:28 PM) ogasawara: Note that being able to set a bug to Triaged and also to set the Importance requires that you be a member of the Ubuntu Bug Control team in Launchpad.
(01:12:43 PM) ogasawara: To learn how to join the ubuntu-bugcontrol team, refer to https://launchpad.net/~ubuntu-bugcontrol
(01:12:56 PM) ogasawara: I'd also like to bring up one last thing to keep in mind when triaging kernel bugs . . . and that's forwarding the bug upstream.
(01:13:09 PM) ogasawara: Before a bug can be forwarded upstream, it needs to be confirmed to exist when running the upstream kernel.
(01:13:22 PM) ogasawara: The Ubuntu kernel team has started building vanilla mainline kernel builds for users to test with.
(01:13:31 PM) ogasawara: See https://wiki.ubuntu.com/KernelTeam/MainlineBuilds
(01:13:44 PM) ogasawara: If a bug exists with the upstream kernel, the bug should be forwarded upstream so that the upstream kernel developers are also aware of the issue.
(01:14:05 PM) ogasawara: Additionally, it may be discovered that the bug is fixed upstream and we should pull in the fix back into the Ubuntu kernel.
(01:14:19 PM) ogasawara: If a bug has already been reported to the upstream kernel bugzilla, http://bugzilla.kernel.org/ , we should make sure we set up an upstream bug watch from the Launchpad bug report to the upstream bug report.
(01:14:38 PM) ogasawara: See https://wiki.ubuntu.com/Bugs/Watches for more information on how to set an upstream bug watch.
(01:14:49 PM) ogasawara: That should pretty much cover the basics of kernel bug triaging.
(01:15:00 PM) ogasawara: As always, feel free to take a look at https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies for more information.
(01:15:14 PM) ogasawara: Now that we've touched on the basics, I'd like to take this opportunity to quickly mention Kernel Bug Days.
(01:15:30 PM) ogasawara: For the past couple months the Ubuntu Kernel Team has been organizing kernel bug days every 2 weeks.
(01:15:40 PM) ogasawara: https://wiki.ubuntu.com/KernelTeam/BugDay
(01:15:51 PM) ogasawara: The goal of each bug day is to triage and fix kernel bugs.
(01:16:00 PM) ogasawara: Typically each bug day also has a general focus, for example suspend/resume bugs.
(01:16:12 PM) ogasawara: The entire Ubuntu Kernel Team dedicates their day to focus on the bug day but we would also appreciate any community involvement!
(01:16:27 PM) ogasawara: Each kernel bug day always has a specific Community section which contains a list of suggested bugs for community members to work on.
(01:16:42 PM) ogasawara: And, it just so happens that we're holding a kernel bug day today!
(01:16:49 PM) ogasawara: https://wiki.ubuntu.com/KernelTeam/BugDay/20090901
(01:17:01 PM) ogasawara: So, if anyone would be interested in helping out, this is a great starting point to begin triaging kernel bugs.
(01:17:16 PM) ogasawara: I'd be more than willing to help anyone get started, just ping me in #ubuntu-kernel on FreeNode after this session.
(01:17:39 PM) ogasawara: Ok, I think now is a good time to stop and take some questions before we move on.
(01:17:51 PM) ogasawara: nareshov> question: i have a wireless card that requires firmware i think (broadcom), how can i test mainline kernels?
(01:18:22 PM) ogasawara: nareshov: unfortunately that is one corner case where you won't be able to use the mainline kernels
(01:18:37 PM) ogasawara: nareshov: however, if you know which firmware is required, open a bug and let us know
(01:18:57 PM) ogasawara: ok moving on
(01:19:09 PM) ogasawara: For the debugging part of this session, I'd like to focus on some common types of issues I run into on a regular basis and how I go about debugging them.
(01:19:37 PM) ogasawara: The first type of issue I see reported rather frequently are update/install issues.
(01:19:56 PM) ogasawara: The majority of these bugs are usually reported via apport and will come in with general debug information already attached.
(01:20:09 PM) ogasawara: The important debug file to look at will be the DpkgTerminalLog file.
(01:20:30 PM) ogasawara: Usually, somewhere near the end of the DpkgTerminalLog file will hopefully be some error messages about what failed during the update/install.
(01:20:42 PM) ogasawara: For example, lets take a look at https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/398036
(01:20:53 PM) ogasawara: falstaff|h> question: i've bug in karmic kernel, which is also in latest mainline kernel. Should I report the bug to the ubuntu kernel bug tracker or upstream? Or both?
(01:21:02 PM) ogasawara: falstaff|h: both would be great
(01:21:30 PM) ogasawara: falstaff|h: we can then link the launchpad bug report to the upstream one
(01:21:33 PM) ogasawara: ScottTesterman_> QUESTION: Today I noticed a bug report where the initial reporter had marked the bug "Confirmed," but there were no other reports to verify the bug.  What's the best way of dealing with this if I cannot confirm the bug report?
(01:21:59 PM) ogasawara: ScottTesterman_: it's best to kindly remind bug reporters to no "Confirm" their own bugs
(01:22:24 PM) ogasawara: ScottTesterman_: unfortunately if you are unable to confirm yourself, the next best step is to make sure they have all the appropriate debug information attached
(01:23:00 PM) ogasawara: ok, back to bug 398036
(01:23:10 PM) ogasawara: After examining the DpkgTerminalLog.txt file I noticed the following errors:
(01:23:23 PM) ogasawara: Merging changes into the new version
(01:23:23 PM) ogasawara:  Conflicts found! Please edit `/var/run/grub/menu.lst' and sort them out manually.
(01:23:23 PM) ogasawara:  The file `/var/run/grub/menu.lst.ucf-new' has a record of the failed merge of the configuration file.
(01:23:23 PM) ogasawara: User postinst hook script [/sbin/update-grub] exited with value 3
(01:23:47 PM) ogasawara: As you can see, this is really an issue with grub which caused the kernel to fail to install.
(01:23:59 PM) ogasawara: Having seen this error message before I knew this was a duplicate of bug 269539.
(01:24:17 PM) ogasawara: Notice that prior to marking the bug as a duplicate I pasted the error message into a comment in the bug report.
(01:25:12 PM) ogasawara: Recall that I mentioned earlier that part of our job as a triager is to educate the bug reporter.
(01:25:26 PM) ogasawara: Pasting this information into the bug informs the bug reporter which debug file I looked at to find the error, what the error was, and why I marked it as a duplicate.
(01:25:42 PM) ogasawara: Now I know some may be thinking, "How am I supposed to know which bug this would be a duplicate of?!?!".
(01:25:57 PM) ogasawara: Well, having triaged many of these types of bugs myself, I took the liberty to write up a DebuggingUpdateErrors wiki for the kernel:
(01:26:09 PM) ogasawara: https://wiki.ubuntu.com/KernelTeam/DebuggingUpdateErrors
(01:26:19 PM) ogasawara: That wiki documents all the common kernel update/install bugs I've come across, what the error message looks like, and what the master bug is.
(01:26:39 PM) ogasawara: Sometimes you won't even want to mark the bug as a duplicate as it could be Invalid.
(01:26:51 PM) ogasawara: For example, if someone ran out of disk space while trying to upgrade, that's not a kernel bug.
(01:27:07 PM) ogasawara: This type of bug is usually indicated by a "gzip: stdout: No space left on device" line in the DpkgTemrinalLog file.
(01:27:21 PM) ogasawara: The kernel has no control over how much free disk space someone has :) So this is just an invalid bug and they need to free up space.
(01:27:35 PM) ogasawara: The next question some may have is "Why can't the errors be detected automatically when the bug is reported?".
(01:27:50 PM) ogasawara: The answer is it can!
(01:28:05 PM) ogasawara: For the majority of the bugs I've documented in that DebuggingUpdateErrors wiki, we've also written an ubuntu-bugpattern.
(01:28:18 PM) ogasawara: https://code.launchpad.net/~ubuntu-bugcontrol/apport/ubuntu-bugpatterns
(01:28:25 PM) ogasawara: If a bug is filed with apport and we've written a bugpattern to match it, the bug reporter will be notified that they do not need to file a new bug and they will be directed to the master bug instead.
(01:28:45 PM) ogasawara: If there is no master bug, as in the example of someone running out of disk space, they are directed to the DebuggingUpdateErrors wiki explaining the issue.
(01:29:04 PM) ogasawara: The ubuntu-bugpatterns are really helpful but some bugs still slip through.  For example, if the error message were in Spanish instead of English, the bugpattern won't catch it.
(01:29:23 PM) ogasawara: Also, if someone didn't use apport to file the update/install bug, I ask them to take a look at https://wiki.ubuntu.com/DebuggingUpdateManager
(01:29:40 PM) ogasawara: The "Debugging Procedures" section outlines the debug files to gather and attach to the report.
(01:30:36 PM) ogasawara: Ok moving on to the next type of issue to debug . . .
(01:30:49 PM) ogasawara: The next common scenario I come across triaging kernel bugs is kernel regressions :(
(01:31:03 PM) ogasawara: First when any regression is reported, it's important that it gets tagged as a regression.
(01:31:19 PM) ogasawara: At the bottom or each bug report's description there should be a "Tags" line and a yellow pencil edit icon to add, remove, or update a bug's tag(s).
(01:31:38 PM) ogasawara: There are usually 4 different regression tags that kernel bugs will use:
(01:31:52 PM) ogasawara: 1) regression-potential - A bug discovered in the development release that was not present in the stable release.
(01:32:12 PM) ogasawara:   For example, right now Karmic is known as the development release and Jaunty is the previous stable release.
(01:32:25 PM) ogasawara:   If someone finds a regression while testing Karmic while we are still in the development phase, this would be tagged "regression-potential".
(01:32:38 PM) ogasawara: 2) regression-release - A regression in a new stable release.
(01:32:50 PM) ogasawara:   For example, when Karmic officially releases and a regression is found, this would be tagged "regression-release".
(01:33:01 PM) ogasawara:   regression-potential bugs could very well become regression-release bugs.
(01:33:13 PM) ogasawara: 3) regression-update - A regression introduced by an updated package in the stable release.
(01:33:26 PM) ogasawara:   For example, if Jaunty released a new kernel update and if a regression were discovered, this would be tagged "regression-update"
(01:33:38 PM) ogasawara: 4) regression-proposed - A regression introduced by a package in -proposed
(01:33:52 PM) ogasawara:   Prior to any updates being released, packages sit in what's called -proposed.  See https://wiki.ubuntu.com/Testing/EnableProposed .  If a regression is found in -proposed, this would be tagged "regression-proposed"
(01:34:11 PM) ogasawara: https://wiki.ubuntu.com/QATeam/RegressionTracking documents these tags in more detail.
(01:34:23 PM) ogasawara: Also see the regression tracker for the existing list of known regressions:
(01:34:31 PM) ogasawara: http://qa.ubuntu.com/reports/regression/regression_tracker.html
(01:34:42 PM) ogasawara: The regression tracker will automatically update with any bugs which get tagged as a regression.
(01:34:53 PM) ogasawara: The kernel team reviews regressions on a weekly basis so making sure they are tagged appropriately will make sure they get on the team's radar.
(01:35:09 PM) ogasawara: The next important part when dealing with regressions is to try to narrow down when the regression was introduced.
(01:35:34 PM) ogasawara: Recall that I mentioned earlier that there are thousands of commits between each kernel release.
(01:35:46 PM) ogasawara: If we just look at Jaunty which released with a 2.6.28 based kernel and Karmic which will release with a 2.6.31 based kernel, there's likely going to be over 37,000 commits.
(01:36:04 PM) ogasawara: Narrowing down those 37,000 commits is going to be key in helping the kernel team quickly find where the bad commit is.
(01:36:23 PM) ogasawara: What I like to do to help narrow down the regression (ie find the bad commit) is to do what we call a rough bisect.
(01:36:45 PM) ogasawara: A rough bisect basically involves selecting a working and non-working kernel and then systematically selecting a kernel in between the two to test.
(01:37:02 PM) ogasawara: Depending if the kernel in between works or not, we adjust the known working and non-working kernel and repeat the process until we can narrow down the working and non-working kernel as closely as possible.
(01:37:23 PM) ogasawara: This is where I like to use the mainline kernel builds since they're already built and ready to test.
(01:37:48 PM) ogasawara: So lets use a scenario that someone says they can suspend/resume just fine using Jaunty.
(01:38:03 PM) ogasawara: They just recently updated to Karmic and are running a 2.6.31-8.28 kernel and their system is hanging on resume.
(01:38:20 PM) ogasawara: First, I like to confirm that the regression is not an Ubuntu specific regression.
(01:38:32 PM) ogasawara: To do this I ask the bug reporter to test the mainline kernel which the current Ubuntu kernel was based on.
(01:38:49 PM) ogasawara: I don't expect anyone know know off the top of their head which mainline kernel an Ubuntu kernel was based on.
(01:39:02 PM) ogasawara: This is why I'd encourage you to use http://kernel.ubuntu.com/~kernel-ppa/info/kernel-version-map.html
(01:39:20 PM) ogasawara: This maps every Ubuntu kernel version to the mainline kernel version they were based on.
(01:39:39 PM) ogasawara: Examining the kernel version map, we see that 2.6.31-8.28 was based on mainline 2.6.31-rc7
(01:39:57 PM) ogasawara: As a result I'd point the bug reporter to the 2.6.31-rc7 mainline kernel build and ask them to test.
(01:40:12 PM) ogasawara: http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.31-rc7
(01:40:20 PM) ogasawara: If they confirm the bug remains, it's a good assumption this was a regression introduced by an upstream change.
(01:40:36 PM) ogasawara: Assuming this is the case, I want to get confirmation that this is still working with the upstream kernel Jaunty was based on.
(01:40:52 PM) ogasawara: Again looking at the kernel version map I see that Jaunty released with a 2.6.28-11.42 Ubuntu kernel which was based on the 2.6.28.9 upstream kernel.
(01:41:13 PM) ogasawara: So I of course ask them to test http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.28.9/
(01:41:29 PM) ogasawara: Then we will have established that 2.6.28.9 is our working kernel and 2.6.31-rc7 is our non-working kernel.
(01:41:45 PM) ogasawara: Now we have to pick a kernel in between, so lets pick 2.6.30
(01:41:54 PM) ogasawara: http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.30/
(01:42:20 PM) ogasawara: We ask the reporter to test this and just for example sake, lets say the bug reporter tests 2.6.30 and notes that it is suspending/resuming just fine.
(01:42:37 PM) ogasawara: So now lets pick a kernel between 2.6.30 and 2.6.31-rc7, say 2.6.31-rc3
(01:42:51 PM) ogasawara: http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.31-rc3/
(01:43:10 PM) ogasawara: And again for example sake, lets say the reporter again says 2.6.31-rc3 works.
(01:43:34 PM) ogasawara: Now lets ask them to test 2.6.31-rc5, http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.31-rc5/
(01:43:49 PM) ogasawara: And agian for example sake, lets say 2.6.31-rc5 still works after they test.
(01:43:59 PM) ogasawara: Now we finally ask them to test 2.6.31-rc6, http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.31-rc6/
(01:44:12 PM) ogasawara: And lets suppose they come back this time and report 2.6.31-rc6 fails to resume.
(01:44:26 PM) ogasawara: So now we've narrowed down the regression between 2.6.31-rc5 and 2.6.31-rc6.
(01:44:42 PM) ogasawara: This is definitely a big help, however there are still 578 commits between -rc5 and -rc6.
(01:45:09 PM) ogasawara: Examining the timestamps for each of those builds, you'll notice that 2.6.31-rc5 was built on 01-Aug-2009 and 2.6.31-rc6 was build on 14-Aug-2009.
(01:45:42 PM) ogasawara: The nice thing is the Ubuntu kernel team also provides mainline daily kernel builds.
(01:45:51 PM) ogasawara: http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/
(01:46:06 PM) ogasawara: So now we can apply the same rough bisect idea but use the daily kernel builds.
(01:46:19 PM) ogasawara: We can then hopefully narrow down between which exact two dates the regression was introduced.
(01:46:40 PM) ogasawara: So just for example sake and to speed things along, lets say we narrowed down the regression between daily kernel build 10-Aug-2009 and 11-Aug-2009.
(01:47:00 PM) ogasawara: Looking at the 10-Aug-2009 build log, http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/2009-08-10/BUILD.LOG
(01:47:14 PM) ogasawara: I see this was built based on commit f4b9a988685da6386d7f9a72df3098bcc3270526 - "Merge branch 'for-linus' of git://git.infradead.org/ubi-2.6"
(01:47:27 PM) ogasawara: Likewise looking at the 11-Aug-2009 build log, http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/2009-08-11/BUILD.LOG
(01:47:42 PM) ogasawara: I see this was built based on commit 85dfd81dc57e8183a277ddd7a56aa65c96f3f487 "pty: fix data loss when stopped (^S/^Q)"
(01:48:02 PM) ogasawara: There are now only 45 commits between these two builds.
(01:48:27 PM) ogasawara: Pointing a developer to examine 45 commits is much more manageable than asking them to examining 37,000+ commits or even 578 commits.
(01:48:53 PM) ogasawara: From that point a developer should be able to post additional test kernels to narrow down the exact offending commit which is causing the regression.
(01:49:14 PM) ogasawara: Unfortunately I have not yet documented this rough bisect process in a wiki yet.
(01:49:28 PM) ogasawara: If anyone is feeling ambitious, feel free to document it and I'll be more than happy to proof read :)
(01:49:50 PM) ogasawara: So I wanted to leave the remaining time to field any questions.
(01:50:03 PM) ogasawara: I apologize if I didn't get to something you were hoping I would.
(01:50:10 PM) ogasawara: I would however point you to our KnowledgeBase that contains lots of good additional debug information.
(01:50:17 PM) ogasawara: https://wiki.ubuntu.com/KernelTeam/KnowledgeBase
(01:50:27 PM) ogasawara: syedam> QUESTION : instead of using a bisect approach cant we see what files were touched in a paticular commit and check that commit and use this with the bisect method
(01:51:29 PM) ogasawara: syedam: indeed each commit outlines the changed set of files, and you could much more quickly do a bisect off of that
(01:51:47 PM) ogasawara: syedam: however that would require having some git knowledge and knowing how to build your own kernel
(01:52:29 PM) ogasawara: syedam> QUESTION: is there an easy way to build and package kernels
(01:53:19 PM) ogasawara: syedam: indeed there are.  I'd take a look at https://help.ubuntu.com/community/Kernel/Compile
(01:54:40 PM) ogasawara: any other questions?  if not we'll end just a few minutes early and I'll turn it over to didrocks.
(01:54:54 PM) ogasawara: bas89> QUESTION: If my system crashes, what are the main signals that the kernel was responsible for it?
(01:55:19 PM) ogasawara: bas89: the most noticeable indication will probably be a kernel panic
(01:55:46 PM) ogasawara: bas89: sometimes it'll get logged to dmesg, but sometimes you'll just see it flashed to your terminal and that's it
(01:56:08 PM) ogasawara: bas89: it's always best to capture as much of the panic as possible if you report it in a bug
(01:56:27 PM) ogasawara: Quarth> QUESTION : Are changes classifyed or someway seachchable by any classification method (like key words or subsystems, layers., sections..)? Real case: i have a problem with my lapton screen resolution, starting with 31-4 runs fine, starting with 31-8 is wrong. Should I use the bisect approach or there's a better way?
(01:57:17 PM) ogasawara: Quarth: if you are familiar with git you could list all the changes based on a subsystem, if not I'd suggest using the bisect approach
(01:57:33 PM) ogasawara: syedam> QUESTION:  are the mainline / daily  kernels built with debugging info
(01:57:56 PM) ogasawara: syedam: unfortunately I don't think so.  I believe there is actually an existing bug report in Launchpad requesting this
(01:58:10 PM) ogasawara: falstaff|h> question: It also happend to me that ubuntu just freezed... Magic Keys worked, can i get the last output of dmesg after rebooting? syslog doesnt contained them..
(01:58:36 PM) ogasawara: falstaff|h: try examining /var/log/kern.log.0 which should contain logs for a few boot cycles
(01:59:44 PM) ogasawara: Ok, I'll turn it over to didrocks who'd going to teach you how to update a package.
(01:59:46 PM) ogasawara: Thanks everyone!
(01:59:59 PM) didrocks: Thanks a lot ogasawara and congrats!