Crash SRU Exception

This document describes the policy for introducing new upstream- and micro- releases of the crash utility into Ubuntu releases. Crash typically has two releases per calendar year, around April and November. This can include minor (8.0.4 -> 8.0.5) and major version updates (7.3.0 -> 8.0.0), and both scenarios should be covered under the testing plan detailed in this page.

About crash

Crash is a self-contained tool that can be used to analyze kernel memory dumps (e.g. created by makedumpfile). It's also able to investigate live systems through e.g. /proc/kcore.

Upstream resources:

• Project page

• Upstream changelog

• Extensions modules

• Contribution Guidelines

Rationale for the exception

Crash is extremely helpful for analysis and RCA of kernel issues. Particularly for HWE kernels, new features could be introduced that change the memory structures being parsed by crash. Keeping crash updated to latest stable versions keeps compatibility for older kernels, while allowing LTS releases to work with kernel dumps from newer Ubuntu versions.

Upstream policy enforces backwards compatibility

Although crash does not seem to have a formal release policy documented, upstream maintainers take extensive care so that the tool always remains backwards-compatible. This ensures minimal breakage and greater stability of the tool, as any updates are usually restricted to bug fixes and to ensure the tool can continue analyzing memory structures of newer kernels. This is documented in the Contribution Guidelines.

Microreleases are not restricted to bug fixes

As there doesn't seem to be a formal release policy documented, there's no criteria of what can be included in upstream releases. Historically, this has involved compatibility patches to new kernels and architectures, regression fixes, cleanup commits, as well as minor quality-of-life features. Considering a new crash release is not necessarily restricted to bug fixes only, we can't always push an upstream release to Ubuntu as per the New upstream microreleases section of the SRU docs. The upstream project also doesn't seem to have a formal testing procedure in place, which we hope to cover for Ubuntu more extensively through the test plans below.

SRU Process

For new releases of the crash utility, the following criteria need to be validated and documented in a public Launchpad bug:

1. Crash must be able to correctly open a kdump-captured image of the system
2. Crash must be able to correctly execute against HWE kernels

Kernel crash dumps should be captured with default parameters for a given Ubuntu release, as this will cover the more general scenario for the tool. Alternatively and with appropriate justification from the SRU requester, crash can be validated against a live system image as well.

The SRU should ensure major supported architectures are working correctly with a new crash version. x86_64 and ARM64 should be considered mandatory; other supported Ubuntu architectures are left at the discretion of the requester. Likewise, we're only concerned that crash is able to open and parse kernel dumps correctly, extensions or specific crash commands are not going to be covered with this test.

Flavors & derivative kernels

The targets above should cover most of the crash users, but it does leave out specific flavors and other derivative kernels like linux-aws, linux-azure, etc. Although there are plans to have full automated coverage of all Ubuntu kernels eventually, these derivative kernels could be considered close enough to their regular counterparts as to not warrant dedicated testing. The reasoning is that the majority of the backports for the flavors are going to have an upstream/mainline counterpart, for which upstream crash will need compatibility regardless. If we keep crash updated and compatible to the latest upstream kernel (and the GA/HWE Ubuntu kernels as explained above), these backported patches should be covered as well. The kernel-team has agreed that these flavors do not need specific coverage, as per this discussion on the kernel-team ML.

Requesting the SRU

The SRU should be requested following the regular SRU process, with additional notes about the validation steps described above. A suggested template for new releases covered under the SRU exception can be found below.

Template

New Upstream release for crash-<version>

[Impact]

This new release contains important bug fixes and compatibility patches for [...].

[Upstream Changes]

TODO: link to upstream changelog, making note of any significant commits like compatibility to new kernel versions or new architectures

[Test Plan]

The test plan from CrashUpdates was followed. Attached are console logs for each covered kernel version and architecture.

[Where problems could occur]

TODO: document any potential issues or risky patches, as per regular SRU process

[Other Info]

TODO: fill out any relevant information to the test plan or the new release

Checklist

• Ensure crash can open dumps for the GA kernel on x86_64

• Ensure crash can open dumps for the HWE kernel on x86_64

• Ensure crash can open dumps for the GA kernel on arm64

• Ensure crash can open dumps for the HWE kernel on arm64

• Attach or link to logs confirming above tests
• Update the list of previous crash updates below

Previous crash updates bugs