Ubuntu Wiki

Please check the status of this specification in Launchpad before editing it. If it is Approved, contact the Assignee or another knowledgeable person before making changes.

• Launchpad Entry: dpkg-lzma

• Packages affected: apt, dpkg, some other large packages

Summary

The Ubuntu alternate cd is running low on space as always and LZMA can compress much tighter than gzip or bzip2, so we intend to allow packages to use LZMA compression as an alternative to gzip and bzip2. Switching to lzma could save us up to 175MB on the alternate cd over what we are currently saving with using gzip and bzip2, depending on which and how many packages we convert.

Release Note

Packages will now be allowed to use lzma compression in addition to the already supported gzip and bzip2 methods.

Rationale

This change is being proposed primarily to save space on the CDs. It would also save bandwidth for downloads from Ubuntu servers.

Just converting the top ten packages alone would save 51,552,804 bytes:

Note: Some packages may not be able to be compressed using lzma due to installation bootstrapping issues.

The trade-off of using lzma is that it is much slower to compress and uses much more memory during compression and decompression. However, lzma is more than twice as fast at decompressing as bzip2, but is still slower than gzip.

Example: openoffice.org-core

The following ODS file has a comparison of the sizes of all packages on the Ubuntu 7.10 Alternate CD compressed with gzip, bzip2, lzma compared to what is currently in the archive:

ubuntu-alternate-cd-lzma.ods

If we were to convert the entire Ubuntu 7.10 alternate CD to each of the following formats it would take the following amount of space:

Use Cases

We would like to fit more packages onto the release CDs.

Assumptions

The primary assumption is that users will have the extra 30MB of RAM (over bzip2) needed to decompress packages.

Design

None (bulk of design work has already been done; this is chiefly deployment)

Implementation

UI Changes

None

Code Changes

The following packages need to have lzma support and currently have it:

• dpkg
  • installing - 1.13.25 (Tue, 2 Jan 2007 00:23:57 +0200)
  • creating - 1.14.0 (Tue, 08 May 2007 11:11:50 +0300)
• apt - 0.7.5 (Wed, 25 Jul 2007 20:16:46 -0300)

However, apt doesn't fully support lzma yet, but this doesn't appear to affect package upgrades. In particular apt-ftparchive contents did not work. I have included a patch that supports lzma except for one function in apt-pkg/acquire-item.cc 'pkgAcqIndex::Failed' that someone more familiar with apt should look over.

apt_0.7.9-lzma.diff

dpkg needs a Depends on lzma.

Launchpad will need to be modified to support lzma uploads and to check that the needed predepends listed in the package.

To build packages, when using debhelper, using lzma compression you need to use: dh_builddeb -- -Zlzma

Migration

To support migrating to using lzma compression dpkg should depends on lzma and packages that use lzma compression should pre-depends on that version of dpkg or newer.

Test/Demo Plan

I tested locally converting all the packages from the Ubuntu 7.10 Alternate CD to lzma compression and tried installing several which 'just worked'.

Future Work

If we also start using lzma for Packages files we could save ~ 70K for main and ~ 430K for the universe Packages file. That might not sound like much but adds up since it is downloaded frequently.

Packages files:

Comments

* "If we also start using lzma for Packages files", These are probably updated much more often than they are downloaded. Perhaps gzip+zsync would be more appropriate than lzma? -- JohnMccabeDansted

**if you use -f then compression time is more comparable with gzip at the expense of size (but still better than gzip) --instigatorirc@gmail.com

CategorySpec