Kernel

Kernel

Discuss How to Improve Overall Boot Performance from a Kernel Perspective

  • How to improve kernel boot time?
    • Grub menu delay - 1st step
  • initramfs - is it possible to reduce the modularity of the kernel?
    • - What are the side effects? Can get too big to load into memory.
  • Which (essential) modules can/should be build in to reduce the number of module loadds
  • Perhaps a fastboot flavors?
  • Balance what's built-in with what's in the initramfs.
  • Hibernation is handled by initramfs
  • 2.6.28 has an option to bootchart kernel internals
  • Focus on use case of average laptop with spinning media
  • Build-in alsa, should give benefit
  • Can't blacklist built-in modules
  • Modprobe is slow and inefficient, but upstream modprobe is improved.
  • Metamodules will have to exclude some drivers that (e.g.) have suspend problems and need to be unloaded/reloaded.
  • Omit hotplug drivers from meta-modules?
  • How long can we defer powernowd? Until after login probably?
  • Action: To get a list of modules that can be built into the kernel
    • - ext3, fan, etc.
  • Readahead optimization is very different for spinning media vs (for example) SSD media.
  • Prefetch adds auditing and accounting to the page cache
  • Possible use audit hooks to map boot for later auditing
  • Action: Scott to look at klibc and initramfs

Having faster boot time by using saved images

  • Looks like TuxOnIce is too intrusive for the Ubuntu kernel, but the grub patch is okay.

Kernel Bugs

  • bugs list all open release targets, hard to tell whether the bug really applies to a release, and hard to list bugs for only one release
  • No way to list all bugs assigned to the people in a group or team
  • Leann's bug list
  • Bugbags may be a solution for some of what Leann's doing
  • want ability to hide bug comments that don't apply
  • want the ability to take a set of comments and attachments which are really a separate problem, and create a new bug with those as initial content. The original bug would have an indication that this happened and pointers to the new bug.
  • Need the ability to lock a bug to prevent reopening or additional comments.
  • How should bug workflow states be set in order to prevent hangups in process? Perhaps to set everything to NEW when it's touched.
  • How do we get bugs nominated for release? Kernel developers don't have the ability to nominate.
  • Bugs are open against (nothing) and not a specific release, so if a developer is working on a bug that's not been nominated and a release is made, it now requires a nomination for the release it was opened against.
  • Proposed solution - automatically target bugs against the current release?
  • Action still required if then later a new release is made, in order to carry that forward.
  • Default to current release, and allow reporter to override.
  • Make an initial guess based on browser string?
  • Search tools in launchpad will improve
  • When will the launchpad janitor be turned on?
    • Once we hit freeze for a release, can we freeze updates from the community, or some way to protect them?
    • The problem is that when bug states are changed, they can fall off the list of bugs being monitored for the release.
  • Set attributes at certain milestones, and disallow some actions on bugs when those attributes are set?
  • Can't negate a tag in a search
  • Should be able to take bug search results, select a subset of the results, and throw those into a bug bag.
  • for the linux package, can we have a template that lists the required information that we need - lspci, dmesg, etc?
    • This is already in the works, and will be here soon.
  • The template sort of exists, but needs refinement, for example make it clearer that the user should run "ubuntu-bug -p linux"
  • We currently build test kernels and deliver them using out of process methods (people.ubuntu.com), we should bring that into the PPA process. Maybe if we could specifiy a subset of packages to build it would be faster. We need to also make sure that PPA builds work for ARM architecture.
  • Ability to use the hardware database, and associate the data with launchpad bugs.
    • This is under way, we should have some API data very soon.
  • Can we weight the importance of bugs based on the pervasity of the hardware among our users?
  • Is there a way to suspend and resume individual devices, in order to test suspend/resume of devices more easily?

ext4 Filesystem in Jaunty

  • Ted believes it's in good enough shape to be used in distro's as an option *NOT* the primary FS.
  • Grub needs work to boot it via UUID.
  • Installer work is needed
  • Agreed to make it available for testing in Jaunty as an alternate FS *not* primary.

Boot Profiling & Making I/O more efficient during boot

  • Boot Performance (AKA run, ubuntu, run!) =
  • 5s pause in Grub
  • initramfs load and expand
  • Goal: to boot in 15 seconds in a normal PC?

    • actually it is: boot a kernel+modules in <4s (Can be done in SSD HD, but in the spinning ones?)

  • compiling drivers in cause their init routines to run; slows booting
  • 2.6.28 has bootchart support for initial kernel boot.
  • There is a size limit on kernel+initramfs that can be loaded into memory
  • modprobe is really slow because it does loads of string stuff to find a module, then loading the module is non-trivial locking wise
    • pciv8086d3945sv0000sd0000magic == CARD

    • modules.aliases converts *v8086v3945* -> iwl3945

    • modules.dep to convert iwl3945 -> /lib/modules/<ver>/.../iwl3945.ko

    • insmod loads those
    • if we cache these answers we are going to save a load of time, and there are very few live answers
  • sReadahead and fast-boot kernel AKA 5 secs boot (Arjan van de Ven )
    • kernel patch to do file load auditing was rejected by upstream - so no other improvement over readahead
  • Detecting modules at 1st boot, then build the modules into the kernel, depending on each PC (Can be done?)
  • OR detect modules at 1st boot, then cache them and load those at sucesive boots.
  • Meta module
    • All modules required on particular HW is linked together into a meta module
  • If new hardware is connected, recache the modules.
  • Modules that should be compiled in:
    • alsa core
    • ext3
    • unix
    • ac
    • fan
    • ata
    • governors
    • Every single hardware driver a given computer needs
  • Get a list of modules that cane very safely be built into the kernel - ext, fan, etc
  • OEM team experiment with metamodules, see what groupings are effective
  • Mine the Hardware Database to get information about common hardware use cases
  • Look at Smolt for hardware info
  • Create a wiki page describing how to use the new kernel boot profiling options
  • Readahead
    • which blocks to read and in what order?
    • prefetch adds auditing and accounting to page cache
    • Can we record load times of f.s. blocks (in jiffies?) - very low overhead.
  • Action items
    • Examine what's needed in the kernel to maintain a list of blocks and audit them later. This will be present in Jaunty as a seed to collect data for improving future releases
    • publish a wiki page to track this
    • get rid of l-r-m and dkms it, would speed up boot. No tempfs, no C compiler (rtg)
  • Next topic: Perform device probing in parallel on different busses
  • Next topic: "Stop the legal fiction of late binding in linux-restricted-modules"
  • Discussion: can we use the kernel audit hooks to save a stream that can be reassembled into a boot description with time stamps?
  • initramfs - at the top, see whether the root fs exists, then use it
  • Action items:
    • Scott to look at klibc and initramfs

Jaunty Filesystems

  • Pushing aufs upstream?
    • An attempt was made, the priority is not high to get aufs into mainline. As long as it works for the users, it's not important to get into mainline.
    • More than a year ago, it was sent to lkml, with few replies that there were minor issues. unionfs people replied negatively.
    • Some things were changed and it was reposted, and it was ignored.
  • Are there any other distros shipping aufs?
    • openSuse includes it, and Suse has a maintainer who makes minor contributions
    • Ubuntu Hardy Heron has a very old version with several bugs and a wrong configuration. (totally wrong, causes crashes)
  • silence normally is really acceptance, so this may well actually be ready to merge, if we just ask
    • Agreed, we can push it if he'd like us to sponsor it
  • Are the configuration options still as complicated?
  • the hardest part to be merged is the architecture??? Has complicated parts. Upstream doesn't like that.
  • He was not aware of the new staging directory in upstream, seems interested in using that.
    • Likes the idea of having help from the community and of getting parts of it into the upstream
  • Monday updates, with changes summarized in email to list, same info in history file.
  • upstream wants to see groups of patches, it seems that currently there is no patches just a release
  • He has very detailed changelogs and email to his users mailing list
  • He does a weekly commit of all fixes to CVS which makes pulling individual patches impossible
  • Canonical needs to put together recommendations on how to help him get this upstream on the ubuntu kernel-team mailing list

Replace apparmor with SELinux

  • kernel:
    • - apparmour is a maintenance burden - selinux is the norm
  • server group
    • - we still want apparmour - we want to support more things, not replace things
  • can we have both
    • - selinux policy is hard, needs a dedicated resource - as of hardy we have both at the same time
  • horse(rick clark)
    • - we don't want to change anything if we don't have to - selinux is ok if we have feature parity
      • - with 'the others' - with app armour
  • apparmour
    • - this is an issue if upstream health is poor
      • - smaller upstream maintenance group these days - patch sets need work, vfs has been issue, less so now - not gonna make .29 for sure - maintainers are supplying us a patch kit for apparmour
      - selinux feature parity is not going to be free either - apparmour is more 'Ubuntu' smaller, thiner, just enough - the upstream community has a central repository for 'my policies' - policies are pretty small - it is possible that a central view of policies as installed could
      • help us
  • selinux
    • - should we be working on this anyhow?
      • - generating some policies to allow people to use this - mostly people are only interested in the server space (protecting
        • network facing apps)
      - policies only work if the disk is correctly labelled
      • - this is tricky to handle in package as its an install time thing
      - debian does have a community here but they are not enabling for lenny - has "audit2allow" - the "steaming pile of policy" problem is that endemic or just a
      • packaging thing?

      - conversion from apparmour -> selinux is not trivial (80% automatable)

  • questions
    • - if apparmour is not ready upstream for next lts, then would we do it? - we are _assuming_ it will be upstream generally - is there any documentation on how to do selinux policy
  • upstream
    • - happy we are carrying it
      • - helps with testing
      - are keen to help us with our problems carrying the change - activly pushing patches upstream

Jaunty-Suspend/Resume/Hibernate

  • Suspend/Resume timing =
    • - we want to get real testing early in the cycle on suspend/resume. want to use our internal people to test the early cycles. something mechanical that anyone in the company can actually do and contribute, technical or not. Open the testing to the world for beta's
  • Discussion ==
    • plan is to use our internal engineering resources to get a base set
    • of tests for suspend resume on all of 'our' machines. to do this via
    • a testing harness preferably reporting back automatically somewhere
    • fast boot is 'good' but super fast resume is 'better'
    • pgraner: my machine is resumed in like 0 time, but then X is 'messing about'
    • horrible messages during resume like removing ff:ff:ff:ff:ff:ff thing
    • (maybe just console loglevel?)
    • aim for 200 suspend/resume cycles. what do we define as successful resume. this could mean resumed to kernel is running, what about
    • networking should we wait for up on network (not in first attempt) suspend power consumption is also interesting. perhaps have one s/r cycle with a 'long' off period
    • can we also use 'subsystem off' to check get fine grained information
    • we are assuming that the acpi wakeup is reliable, this is reasonable
    • and we are going to have to assume so
  • Methods for suspend/resume
    • acpi scripts
    • pm suspend scripts

    • DeviceKit-power

      • not available at this point, for future cycles
  • We should remove the stuff we don't need just install it for testing should we really really really need to use them
  • triggering suspend should occur how --
  • jaunty actions
    • script for testing resume
      • timing information
      • pmtrace
      • rtc magic for failure
      • battery consumption in suspend
      • pgraner has a lot of additional instrumentation also
    • appport lashup for bugs on resume failure (maybe success)
    • can we figure out how apple/vista does the suspend and suspend
    • 'becomes' hibernate
      • those who have access to these 'alien' platforms can see if we see
      • anything
    • review the 'how to diagnose and fix' resume problems
    • consider stripping out the acpi support scripts
      • who the heck do we talk to to fix those
    • kernel: to review /etc/acpi and see what can be stripped and report
    • to ubuntu-devel
    • swap on file, can we do anything here to help in the process
  • testing script
    • What can we/should check:
      • 'we are up'
      • networking is working
      • is the screen on (can we tell)
      • is the backlight on (can we tell)
      • gnome-power-manager -- knows much goodness

Jaunty Kernel Tree Managment

  • We maintain a separate tree for each release, created from the previous tree.
  • The goal is to minimize deltas in the trees.
  • We review the delta and carry a subset of patches to the new tree.
  • By rebasing and losing patches, we lose information about the heritage of the trees.
  • Rebasing is the right thing to do because understanding our delta is the most important thing.
  • By moving our patches to the top of our tree, we can manage them and track them well.
  • We need to communicate to the community better why we rebase rather than follow a head.
  • Need better information in the wiki - mentioned in kernel devel pages??
  • Why rebasing helps us:
    • Dapper - we put patches in, and over time they get lost in the devel history.
    • At times we may patch on top of a patch that went into dapper, etc. After several releases, the patches become dependent on each other, and it's difficult to manage. Rebasing resolves these into a new set of patches at the top of the tree.
    • When we bisect to find a bug or regression, we sometimes end up at a patch in the upstream, and it turns out that the bug wasn't in that upstream patch, but our patch that we carry broke the upstream patch.
  • is git a RCS or a patch queue manager??
    • We use it because it's the upstream tool. We use it more as a patch manager.
    • Part of our workflow that drives our needs is that we create and carry patches until it comes out of upstream. In the process of rebasing, these patches fall out, and we drop them. If we merge, this folding doesn't happen and it looks like we are carrying two patches.
  • Action Item: Discuss and decide whether we need to deliver historical changelogs
    • We do maintain history of how we got to a tree, how it was created, which patches applied and were carried.
  • Why don't we been a branch for each feature set we work on?
    • Becuase we aren't the mainline tree
    • Because so much of our work involves cherrypicking patches from mainline
  • post-release tree maintenance
    • Security - 35-40 CVEs - we maintain five trees
    • It's very quick to cherry-pick upstream patches.
    • we generate the correct formats to submit to upstream.
    • for outside communities and customers, we should encourage them to develop against upstream, and then we can merge the commits back into our tree. This should be relatively easy since we don't carry large changes against the upstream tree.
  • Should we be more careful to put all patches out on the mailing list, during the rc cycle, a lot of changes never hit the mailing list.
  • Is there more information we should be putting in the commit?
    • Provenance is important
  • At what point should we require a bug per commit?
    • It's an extension of out current process.
    • We will do this when we're out of the rc process and have declares our tree stable
  • Action Item: Andy will investigate doing this using commit templates and commit hooks
  • Problem - we rebase from (hardy), then we add to both hardy and the new tree. Then there's no good way to tell which patches have been applied to each of these compared with the other. One solution would be to tag branches at the point when we rebase out of them.
  • At the point we rebase, we need to create a list of what was dropped, why, and publish this.
  • Action Item: Take the email that Tim sent about the last rebase, put it in the wiki, and make it a template as part of the rebase process.
  • Action Item: Pete to create and populate calendar that communicates when we take the major actions on the kernel process. Ben knows about the calendar for Hardy
  • Action Item: Stefan to publish a schedule of kernel uploads
  • Upstream Stable updates - we're getting hundreds of patches every week.
    • Can we merge stable updates? It wouldn't affect bisecting.
  • We cherry pick all updates from stable in order to make sure that everything applies perfectly.
    • Agreed that we should continue to cherry pick.
  • When releasing we tag releases, and later we move them. Why?
    • We retag so that even those the tags aren't historically correct, it allows the changelog to be correctly populated from the tag.
    • Agreed that we should continue to do this.
  • To do an ABI bump, there are multiple things that have to be done. The process around this is not well documented.
  • Action Item: Andy will review this wiki page and consult with Ben and Tim to make it clearer
  • We keep packages in separate trees, can we put (for example) the backports in our main tree?
  • Can we put two different packages in the same tree? (specifically lbm)
    • No, there are good reasons for this to be this way.
    • If we change something trivial like a PCI ID, we shouldn't have to upload and build the entire kernel.
  • We should be encouraging all maintainers of things we carry in /ubuntu to get them into GregKH's staging tree, and for nonresponsive maintainers we should be sending them up ourselves. Greg will take anything that compiles, and that way it gets tested against tip.
  • How do we decide which drivers to deprecate, and how do we publicize this and get feedback?
  • Action Item: review everything we're carrying in the session tomorrow, whatever is left, we'll contact the maintainers. If they don't respond, we'll drop it. We publish a deprecation notice of the things we intend to remove, and take feedback and use that to make final decisions.
  • What do we do with trees when we reach end-of-life? Should we push them all into one tree, with a branch for each release?
    • From dapper forward, we will do this.

Jaunty Central Regulatory Domain Agent (CRDA)

  • CRDA is an upstream initiative to consolidate regulatory info into one database
  • Solves the module params to designate what regulatory domain your in i.e. USA vs. EU
  • Needs to be packaged and distributed in Jaunty
  • rtg to investigate

Jaunty Kernel Team Process

  • This was a white board session outlining critical points in the cycle where specific actions needed to take place.
  • Session not about SRUs, or about security updates, etc
  • From the time we go to GA, when do we rebase, when do we stop that, bug prioritization, How do we know what is going into the release?
  • The purpose is to review and document the process.
  • Start at the end of a release -
    • just before release, enter quiescent state, no longer updating kernel
    • release + 1w, new tree is opened
    • picking up -rc's for the proposed kernel base and rebasing 'Ubuntu'
    • kernel onto that
    • first user space sync (discover and enumerate)
    • release + 1m, alpha1
    • release + 6 weeks, UDS occurs
    • UDS effectively defines the release (release driver lockdown)
      • review of drivers/ubuntu/config options etc, kernel version etc
        • things going away in N+1 are reported for N
        • what we are going to base off etc is decided and reported
    • release + 2m hopefully somewhere here the final kernel releases
    • release + 2m alpha2
    • lots of alphas one per week
    • final user space sync, monitors at alpha rel points
    • release + 2m + 6w beta freeze
  • userspace syncs
    1. when we take our first kernel update
      • patch review
        • sauce
        • ubuntu modules
        • third party drivers
      • pushing upstream
      • config options
    2. alpha2
      • major things need to be matched at this point
    3. slightly before beta freeze
      • this is a final check the things from alpha2 where done
  • Vanilla kernel packages, one off of the stable (.y) tree and one off of the mainline tree

Maintaining the ARM Kernel

  • Why do we have such a huge delta in the OMAP tree?
    • a lot of code which is not very clean (and so no upstream)
    • also a lot of dsp code (secret? it is a bad shape)
    • we also need a 'workspace' when getting stuff working
  • How does the development flow work?
    • The OMAP stuff goes through rmk ARM mailing list, and drivers go through their associated mailing lists.
    • They are planning various branches for drivers that are not yet in the upstream.
    • Their trees are rebased more or less on every rc, more often if new omap code is upstream
  • Suggested that we cherry pick the branches we need. for example, pull the hsmmc branch, and incorporate that, but the patch sets will still be quite large.
  • How many branches?
  • two bsp branches, they don't share much code.
  • How many flavours will we carry?
    • we'll need different for omap2 and 3
  • dsp bridge code should probably be a separate module.
    • (how do we feel about making a separate package for it? does that make any sense to us if its in the same tree anyhow)
    • (is this even open source, can it even be in our tree?) The bridge is open source. and so is a version of the dsp compiler.
  • drivers are the only obviously 'worrying' parts of a merge:
    • bluetooth
    • cbus
    • omap-rng.c
    • crypto
    • dspgateway
  • There are no open source use cases for the DSP code, only closed codecs.
  • There are no drivers that are shared with x86, so minimal risk of regressions to our existing tree.
  • Seems that preliminary plan is for us to rebase our tree as normal onto each of the -rc's and _then_ merge the omap-sane tree against the same -rc.
  • we use a -build branch which is a last second merge from the ubuntu topic and the omap topic to make a real build
  • these will be attached to the tags

Jaunty Kernel Feature Review

UDSJaunty/Report/Kernel (last edited 2009-01-12 20:11:17 by nc-65-40-95-196)