Rolling HWE Stacks for 16.04

This is a spec that has been implemented in 2016 (4 BP - 4 years before pandemic). No further actions required to be completed for this spec, and all releases after 16.04 LTS continue to follow this model.

Background

https://wiki.ubuntu.com/Kernel/LTSEnablementStack

We first introduced the idea of a HWE Kernel in the Ubuntu 10.04 LTS release, Lucid Lynx. The HWE kernel was derived and vetted from a newer kernel shipping in a subsequent interim release of Ubuntu. These HWE kernels were released in the LTS point releases as a means to enable newer platforms and components which required functionality delivered in these newer kernels. Both desktop and server images were seeded with the HWE kernel by default in the point release images. In addition to an updated HWE Kernel, for desktop users, an updated X Stack was also delivered. Beginning with Ubuntu 12.04 LTS, Precise Pangolin, it was agreed that users could remain on a HWE stack (ie HWE Kernel + X Stack) until the subsequent HWE Kernel from the next LTS was introduced in the images of the 5th point release of the LTS. Users were then required to upgrade to this final HWE Stack in order to remain supported with security updates and ongoing bug fixes. An example of this is depicted below:

Include: Nothing found for "^This is a distilled view of our 14.04.x Ubuntu Kernel Support Schedule."!

Include: Nothing found for "= 16.04.x Ubuntu Kernel Support ="!

Proposal

Beginning with the Ubuntu 16.04 LTS release, Xenial Xerus, the Kernel and Foundations team proposed moving to a slightly different model for maintaining HWE kernels. Instead of supporting different HWE Stacks in parallel until we reach the final 5th point release of an LTS, we would like to roll users forward incrementally as the newer HWE Stacks are introduced, ie. instantiate a rolling HWE stack. This is best described with the following diagram:

Include: Nothing found for "^This is a distilled view of our 16.04.x Ubuntu Kernel Support Schedule."!

Include: Nothing found for "= Ubuntu Kernel Support ="!

The Release Team has also agreed that for 16.04 server images, they will offer both the GA and HWE Kernels. The server images will default to the GA kernel and server installs could then optionally opt into the HWE Kernel stream. Desktop installs will continue to offer the HWE Stack option only and default to it.

Justification

The purpose of the HWE kernels has always intended to be used as a mechanism to deliver support for newer platforms and hardware components requiring functionality provided in a newer kernel. This has allowed Ubuntu to support customers/users with newer hardware but still under the umbrella of an LTS release. Moving to a rolling HWE stack model has no impact on delivering this newer hardware support to users in an LTS.

The HWE Stacks were also meant to be used as a stepping stone to move a customer/user to the next LTS release. The HWE stacks were never meant to be supported for the entire life of the LTS release. Doing so would require some teams to treat every release as an LTS release with multi-year maintenance commitments. We simply do not have the resources and staffing to do so and thus have always required the mandatory upgrade to the final HWE stack delivered in the 5th point release images. Moving to a rolling HWE stack model reinforces the notion of migrating a customer/user to the next LTS release.

The previous HWE stack model required we maintain kernel and Xorg releases in parallel, often well past their natural supported life from the interim release for which they were derived. This was a maintenance burden on teams. Teams would have to incrementally pick up additional package maintenance, and at the worst case, teams were having to maintain up to 4 stacks in parallel. Moving to a rolling HWE stack reduces this burden and allows teams to maintain at most 2 stacks at any given time. It would also never require teams to maintain a stack past its natural supported life in the interim release for which it came from.

Since the previous HWE stack model required we support kernels for longer periods of time, it also meant we had to take on the additional upstream stable kernel maintenance for a given HWE kernel. This again was a staffing burden which is eliminated if we move to a rolling HWE stack model.

Maintaining multiple stacks in parallel also increased the complexity of packaging, testing, and supporting clean upgrade paths. Moving to a rolling HWE stack model greatly simplifies this.

It should also be noted that regardless of sticking with the previous model or moving to a new rolling HWE stack model, users in both cases will be required to upgrade to the final HWE stack delivered in the final point release of the LTS.

Implementation

We will move to a rolling HWE stack model beginning with the Ubuntu 16.04.2 LTS point release. For server images, both the GA and HWE kernels will be made available, however images will default to the GA kernel and offer the latest HWE kernel as optional. Desktop images will continue to only offer the latest HWE kernel beginning with 16.04.2. In addition to the GA path, there are 2 alternative paths from which users can consume HWE Stacks for 16.04. We will document all 3 below:

• ga-16.04
• hwe-16.04
• hwe-16.04-edge

ga-16.04

Description

The Ubuntu 16.04 LTS release ships with a standard Ubuntu v4.4 kernel. It is commonly referred to as the GA kernel and is supported for the 5yr support window of the LTS. As noted above, both the GA and HWE kernels will be made available for server images, however, server images will default to the GA kernel and offer the latest HWE kernel as optional. Desktop images will only offer the latest HWE kernel beginning with 16.04.2.

Security updates and bug fixes provided as SRU’s to the GA kernel still remain under the GA umbrella for the full 5yrs of support. Stable release updates are still considered ga-16.04 despite them not being shipped in the ISO. For example, 16.04 released with a 4.4.0-21.37 Ubuntu kernel, but it has since updated to 4.4.0-38.57 (at the time of this writing). 4.4.0-38.57 is still considered ga-16.04 despite it not being shipped in the ISO.

In the case of custom kernels, we have communicated to our partners (eg. Amazon, Google, Microsoft), that they can customize their upgrade cadence. They are welcome to adopt our rolling HWE model or not, it's really at their discretion given it's their custom kernel. At this time, we do not plan on offering both GA and HWE variations of custom kernels for partners. None have expressed an interest for this. For awareness, custom kernels will follow the naming convention of flavors, eg. linux-aws.

Packages

The exact meta package names are:

• linux-generic
• linux-lowlatency

Update Schedule

The GA kernel will receive continuous stable release updates to address security vulnerabilities and fix general bug escalations. It will continue to remain a v4.4 based kernel for the 5yrs of support for the LTS.

hwe-16.04

Description

This represents the path where HWE Stack users on Ubuntu 16.04 LTS will automatically upgrade to newer HWE Stacks until reaching the final 18.04 HWE Stack in Ubuntu 16.04 LTS. Users will then remain on this final 18.04 HWE Stack for the remaining supported life of the Ubuntu 16.04 LTS. If the user fully upgrades to Ubuntu 18.04 LTS, they will remain on the GA Kernel delivered in Ubuntu 18.04 LTS and will not continue rolling forward on HWE Stacks delivered for 18.04. This is the path that users will be put on if choosing to install the HWE Stack from the 16.04.2 or newer LTS point release.

Packages

The suggested exact meta packages names are:

• linux-generic-hwe-16.04
• linux-lowlatency-hwe-16.04

These meta packages will resolve to the appropriate linux-image-<flavor>-hwe-16.04 and linux-headers-<flavor>-hwe-16.04. See example diagram below. In the event we need to provide extended support for customers for a specific HWE kernel, we can roll out a linux-<flavor>-lts-<release> indirect meta package into a private PPA.

Update Schedule

We will roll to the newest HWE Stack offering around the time of the point release introducing that HWE Stack. This is approximately 3mo after the original series is released. As an example, the 16.04.2 point release should take place in January 2017. At this time the linux-headers-generic-hwe-16.04 and linux-image-generic-hwe-16.04 would point to the new v4.8 based kernel.

As standard practice, we have also agreed with the Archive Admins that we will deliver the target kernel for a point release into -proposed ~2 Kernel SRU cycles in advance of the point release shipping date. This will allow daily images to be spun appropriately leading up to the point release and an opportunity to land any necessary bug fixes.

hwe-16.04-edge

Description

This represents the path that provides users early access to the upcoming HWE Stack that is to be released next for the Ubuntu 16.04 LTS. The aim is to provide this early preview in order to allow users to test the upcoming HWE Stack prior to the automatic upgrade taking place. This should help ensure a smooth upgrade and transition. If users wish to remain on this path, they will continue to be provided the latest edge HWE Stack until reaching the final 18.04 HWE Stack in Ubuntu 16.04 LTS. Users will then remain on the final 18.04 HWE Stack for the remaining supported life of the Ubuntu 16.04 LTS. If the user fully upgrades to Ubuntu 18.04 LTS, they will remain on the GA Kernel delivered in Ubuntu 18.04 LTS and will not continue rolling forward on HWE Stacks delivered for 18.04.

Packages

The suggested exact meta packages names are:

• linux-generic-hwe-16.04-edge
• linux-lowlatency-hwe-16.04-edge

As with the hwe-16.04 path, these -edge meta packages would resolve to linux-image-<flavor>-hwe-16.04-edge and linux-headers-<flavor>-hwe-16.04-edge

Update Schedule

As mentioned, the kernel version provided by the hwe-16.04-edge path will provide early access to the newer kernel version up to ~6mo prior to the hwe-16.04 path receiving the same.

We will introduce the newer edge kernel at each point release. For example, 16.04.1 released in July, 2016. At that time, hwe-16.04-edge would point to the Yakkety kernel currently undergoing development for the 16.10 release. This will allow for a maximum ~6mo baking period before being officially delivered and supported in the 16.04.2 time frame. Once hwe-16.04-edge and hwe-16.04 both point to the final Yakkety HWE kernel version they will remain on that kernel until the 16.04.2 point release takes place. Immediately following the 16.04.2 point release, hwe-16.04-edge will again move to point to the Zesty kernel undergoing development (ie. the eventual HWE kernel for 16.04.3).

Backwards Compatibility

The HWE kernel may introduce user facing behavioral changes associated with a kernel version change. That may be intentional (or not) but we will make a best effort to provide backwards compatibility in order to not disrupt user expectations. In the event where userspace changes could be SRU’d to accommodate the intended behavioral change, that would be preferable so that the HWE kernel could track as closely as possible to the interim kernel release from which it is derived.

For an example here are 2 bugs we've found in the last 2 weeks with 4.8 entrance into yakkety:

• unintended: http://pad.lv/1628336

• intended: http://pad.lv/1493188

Kernel Livepatching

https://www.ubuntu.com/server/livepatch

For clarity, the Canonical Livepatch Service is only available and supported against the generic and lowlatency GA kernel flavours for 64-bit Intel/AMD (aka, x86_64, amd64) builds of the Ubuntu 16.04 (Xenial) and 18.04 (Bionic) LTS releases.

Outcome

This will go into effect beginning with 16.04.2.