ARMKernelVersionAlignment

Revision 10 as of 2010-06-02 08:33:15

Clear message

Summary

Establish a process and work flow to integrate together constantly evolving kernel trees from various independent sources in order to merge the result into the main Ubuntu kernel package, while ensuring an easy path towards integration into the upstream kernel tree.

Release Note

Ubuntu 10.10 is made of a common kernel source tree across all supported architectures and target, including ARM.

Rationale

Unlike for X86, what we generally refer to when mentioning "ARM" is pretty eclectic. While ARM Ltd has defined and standardized the ARM instruction set, the ARM licensees (aka vendors i.e. those who actually produce chips) have integrated that technology into wildly different SOCs. Therefore, there isn't such thing as a "common ARM architecture" as we mean it when talking about the "X86 architecture".

Those different ARM vendors have used their own set of IP blocks around the ARM CPU core, such as timers, interrupt controllers, memory controllers, IO peripherals, etc. Even the MMU architecture has seen some variations between different vendors and different revisions of the ARM architecture. There is some on-going work at ARM Ltd to standardize more pieces of an ARM system into a common specification. But the ARM systems we have to deal with today come from different vendors with considerable differences, almost like different architectures when it comes to software support.

The chip vendors usually provide Linux support for their product. The problem of course is a diverse set of large patch series for separate SOC families, often based on different kernel versions, and with varying levels of implementation quality. Those different SOC families have their own on-going development and particular issues too.

Until now, each release of Ubuntu on ARM has used different kernel source packages, sometimes even using different base kernel versions, to cope with the kernel discrepancies between all supported vendors. But this is causing no end of support, maintenance and QA issues.

Assumptions

This specification is not about a feature per se. This is rather a proposal for a development process which has minimal user visibility. While the end result should be the same to the user, this is meant to help Ubuntu kernel developers.

Design

Distributed Development

Following a model similar to the one used by the upstream kernel developers and maintainers, it is desirable to distribute the work amongst people with access to specific hardware, and make them responsible for smaller isolated parts. This is easier for those individuals to work in close relation with the vendors, monitor and also participate to related development taking place in the relevant communities, if the autoritative reference for on-going development work remains isolated and separate from other work.

Continuous Integration

At the other end of the system, it is necessary to integrate all the supported kernel "architectures" together, and perform that operation on a regular basis. This allows for:

  • Find out about merge conflicts early and determine a strategy to avoid/solve them.
  • Provide a unified kernel tree for testing.
  • Allow for a kernel package to be available in the distribution archive containing the latest developments prior the freeze date.

While consolidation depends on the merge of various subparts, it is important not to prevent those subparts from moving on along with the external communities they're part of in parallel to our consolidation needs.

Implementation

Let's define a few roles.

Feature Trees/Owners

The feature owner is someone with responsibility over a Git tree containing a particular feature.

For example, one person should handle everything OMAP related in a separate tree, another should handle Freescale related support (and the different i.MX flavors be split as well if relevant), yet another tree for Marvell, etc.

Generic ARM features (such as stack protector support) should go into yet another git tree. And so on. As long as those splits are logically made, and that the support being worked on is isolated and autonomous e.g. the OMAP tree should have all but only the patches needed to boot the target platform and depend on nothing else (no generic Ubuntu features from the main Ubuntu kernel tree but just the base kernel tree from kernel.org).

If the task is trivial enough then one person may "own" multiple trees. But there must be only one owner per tree. That doesn't mean that only this one person does all the development work though. But it is that person who should manage the patches that goes into that tree, either from himself or others. This means sorting out vendor patches, cleaning them up, forward porting them to the kernel version we're targeting (which before the Ubuntu kernel freeze would mean latest from kernel.org), etc.

In parallel to the Ubuntu kernel freeze and release process, the individual feature trees should continue moving on as they should eventually be submitted for upstream merging on the upstream merge schedule. So it is important that they remain clean and genuine as much as possible. The idea is to always keep some authoritative tree for separate features available, either to serve as a component to build the Ubuntu kernel source package and/or to have ready for a merge into upstream.

If some vendors already have a clean tree which meets the Linux community standards, and they're proactive with the community and pushing their patches upstream, then we should merely have to track their tree directly. Otherwise it might be necessary to do a minimum of janitorial work ourselves, and consider leading some effort to have that work merged upstream. Determining how to get optimal collaboration from vendors should be determined.

Feature owners may carry additional branches in their tree as they see fit to improve the work they do on their master branch. This may include other people's feature tree, or the master Ubuntu kernel, etc. This is often useful to generate diffs, do local merge tests, or any other tasks as they see fit. The master branch must be clean and predictable though.

Also, if some generic subsystem is required for the implementation of some features (let's say "consolidated power management" for example), then it is preferable to have it split into separate branches as much as possible not to "contaminate" the master feature branch. For example, a local tracking branch may be used to pull the generic PM stuff, and some OMAP_PM branch may be created to merge both the master feature branch and the generic PM branch together before adding specific OMAP PM changes on top. But this may get a bit complex and discussing such layouts is best postponed for when/if this is actually required.

Merge Tree/Owner

Now, there is someone whose task should be continuous integration of all the ARM tree and the main Ubuntu kernel tree together in a "volatile" unified Git tree.

This merged tree is called "volatile" because it is thrown away and completely rebuilt on a regular basis. Therefore it must be used only for testing purposes and no development must ever be based on that tree. It is also possible to tag those volatile tree results with a date and keep them around for forensic purposes (e.g. this bug wasn't there in last week's merged tree). The optimal remerge frequency has to be determined.

The purpose of this is many folds.

Solving Merge Conflicts

Finding out about merge conflicts should be done as early as possible, even when the merged code is still a work in progress. This should be done by the merge owner, as feature owners should remain focused on their own feature set, and asking each of them to test-merge everyone else's tree won't scale.

Because those feature trees should be independent, in theory there shouldn't be any fundamental merge issues. If this proves to be problematical then a better feature tree split should be considered.

Unified Kernel Tree For Testing

The feature tree owners should be responsible for making sure that their tree actually works on the target they own (either by testing it themselves, or mandating someone else). They should also be responsible for the testing of this unified kernel tree on their own target.

It is important to isolate bugs created by such a merge from the other bugs and take appropriate action. In the case of a merge related bug, the merge owner should be involved in the isolation of the problematic merged feature and coordinate a solution with the concerned feature owner(s).

Constant Availability of a Kernel Package

This unified kernel tree should allow for a kernel source package to always be available in the distribution archive with the latest developments, especially before the freeze date. Once the freeze date has come, this consolidated tree is remerged but without the main Ubuntu kernel tree this time, as the goal is to have the main Ubuntu kernel tree maintainer to merge this consolidated ARM tree instead.

BoF agenda and discussion

Kernel Version

With the current upstream cadence we are expecting v2.6.35 to release mid to late september. With a likely Maverick kernel freeze date at the end of September, we are therefore unable to take an early v2.6.36 release.

The decision therefore is v2.6.35 for Maverick.

ARM VERSIONS

Need to try and get the vendors on the same level, at v2.6.35.

For OMAP3 we are expecting to have a small delta, and hope to get that as a merged branch (in master) and use that as reference platform.

OMAP4 support in 2.6.35 will be there, but very basic, not enough for a full ubuntu stack. There is a fair pile of patches pending for upstream. Likely we will have an OMAP4 enablement branch to allow the full stack to be applied. It will beed backports from v2.6.36.

Most of the ARM bugfixes fixes are already scheduled for v2.6.35, there are some features coming but not significant for us.

Freescale: they are currently planning on v2.6.34. they have timing constraints due to new silicon appearing around v2.6.35 release date

Marvell: v2.6.32, mostly upstream from v2.6.34. It is therefore possible v2.6.35 will just work for Dove. Still may be some patches needed for more than 'basic' support; display is missing.

Samsung: currently targetting v2.6.34 and have some 60 patches on top.

Smoothstone: missing the v2.6.35 window, a fairly large patch stack

Get the message out that we want to get proposed patches for ARM merging around v2.6.35-rc2


CategorySpec