Interdiff

When packaging new upstream versions of software, debdiff is often not sufficient to represent the changes in the packaging when requesting sponsorship, perhaps because the new upstream source includes additional binary files (images, sounds, etc.), or perhaps because the reviewer is overwhelmed by upstream changes, and cannot effectively focus on the specific work being sponsored.

Luckily, the patchutils package has a few utilities which can be used to support a review process in this situation: interdiff, combinediff, and filterdiff. interdiff generates a new patch that represents the difference between two patches. combinediff merges two patches without reference to the modified source, and filterdiff allows the extraction of only part of a patch.

There are two types of interdiff file: a "full interdiff", which is typically compressed, and is the normal means of requesting sponsorship, and a "consolidated interdiff", which is human readable, much like a debdiff, but not supported for automatic extraction by the available tools.

Preparing an interdiff for review

Firstly, because this will be a new upstream version, it is essential that the packaging provide an automated mechanism for downloading the upstream distribution. Be sure to provide an accurate, tested watch file, and if repacking is required, a working get-orig-source in debian/rules.

As with a debdiff, it is important to review the changes to make sure they are all intentional. Given that the full interdiff isn't easily human readable, it's all the more important to get the package right: it's a lot harder to edit the target file after it has been prepared. To check the changes, use the consolidated interdiff generated with:

interdiff -z -p1 packagename_version-revision.diff.gz packagename_newversion-newrevision.diff.gz > check.interdiff

If everything looks good, prepare and compress a full interdiff with:

interdiff -z packagename_version-revision.diff.gz packagename_newversion-newrevision.diff.gz | gzip --best -c - > packagename_newversion-newrevision.interdiff.gz

Before presenting the interdiff to a sponsor, it is a good idea to make sure it is correct, and sufficient to reconstruct the target package. A minimal sanity check is the following:

$ combinediff -z packagename_newversion-newrevision.interdiff.gz packagename_version-revision.diff.gz > test.diff
$ interdiff -z test.diff packagename_newversion-newrevisision.diff.gz
$ rm test.diff

This should produce no output.

Also, test to make sure that the sponsor will be able to easily collect the appropriate upstream source file, as follows:

$ dpkg-source -x packagename_newversion-newrevision.dsc
$ rm packagename_newversion.orig.tar.gz
$ cd packagename-newversion
$ debian/rules get-orig-source || uscan --force-download

This should restore a working orig.tar.gz.

Attach the tested interdiff to the upgrade bug, and subscribe the appropriate sponsors team for review. Note that additional information (e.g. upstream diffstat, changelog deltas, etc.) may also be required for certain upgrades.

Debian Exception

Ubuntu has a special relationship with Debian, and shares a large volume of software. In order to ensure that patches between Debian and Ubuntu remain compatible, if the upstream version being packaged is available in Debian, a debdiff against the latest Debian package should be submitted for review and sponsorship, in lieu of an interdiff.

Reviewing an interdiff

The main differences between an interdiff review and a debdiff review are that the initial file presented for review is not in a conveniently human readable format, and that the reviewer must collect the updated upstream source as part of the review process. The following procedure will result in a human readable review file, a candidate source package, and an unpacked package source directory, much as one has when comparing an existing source and a debdiff.

Procedure

1. Download the existing source package

$ apt-get source packagename

1. Generate a new candidate diff.gz

$ combinediff -z packagename_newversion-newrevision.interdiff.gz packagename_version-revision.diff.gz | gzip --best -c - > packagename_newversion-newrevision.diff.gz

1. Prepare a consolidated interdiff
   • The full interdiff is rather hard to read, but combinediff doesn't support hunk-splitting, so it is required in order to construct the new diff.gz. For visual inspection, the following command generates useful visual review output:

$ interdiff -z -p1 packagename_version-revision.diff.gz packagename_newversion-newrevision.diff.gz > review.interdiff

1. Download the upstream source

$ filterdiff -z -i 'packagename-newversion/debian/*' packagename_newversion-newrevision.diff.gz | patch -p0
$ cd packagename-newversion
$ chmod +x debian/rules
$ debian/rules get-orig-source || uscan --force-download
$ cd ..
$ rm -r packagename-newversion

1. Prepare a package candidate

$ tar xzf packagename_newversion.orig.tar.gz
$ cd packagename-newversion
$ zcat ../packagename_newversion-newrevision.diff.gz | patch -p1
$ debuild -S -sa -us -uc

Build, test, and sponsor as usual.

Additional concerns when sponsoring new upstream versions

• Check the upstream changelog against the local changes to make sure there's not an unhandled change or accidentally dropped patch.
• Check to see if there has been a licensing change upstream, and if so, verify that the new license is suitable for inclusion.

Automation of the above procedure

There is work underway to script the above procedure, which is expected to significantly reduce the burden on sponsors for this type of review, and improve throughput on the maintenance of new-in-Ubuntu packages.

Handling packages with upstream debian/

If the target package has debian/ in orig.tar.gz, and cannot generate a shell debian/ directory to download the original source, try a more restrictive filterdiff to extract the target files, or review debian/rules, debian/watch and/or debian/copyright for the existing package, to discover a download source. Further, please ask the packager to coordinate with upstream for separate distribution of the debian/ directory.

Handling missing upstream information

If the target package has neither a get-orig-source rule or a watch file, the location from which to download the orig.tar.gz may be shown at the top of debian/copyright. If this is an Ubuntu-local package please request the addition of this information to assist with future maintenance.

If none of the above are present, please request an updated interdiff for review, including some guidance on how to acquire the upstream software distribution.

Deprecated new upstream version review techniques

diff -urn debian/

Whlie it is acknowledged good practice to maintain all package variance within the debian/ directory, not every package does so. A review of only the Debian directory may miss patches previously only in the diff.gz. Similarly, such a review may miss patches applied in the new diff.gz. The consolidated interdiff is similarly easy to read, and ensures that patches will be encountered wherever thay may be found.

trusted orig.tar.gz

While there have been no reported issues with incorrect tarballs to date, it is appropriate to verify that a provided diff.gz works on an upstream tarball (or known derivative thereof). This serves as a means of ensuring that the package does not contain any unexpected changes, and makes the task of packaging the next upstream version simpler, and less likely to silently drop changes. By preparing a new orig.tar.gz during the review process, the sponsor is ensured that the diff.gz is complete, and sufficient for the package.

Deprecated new upstream version review processes

Prior to the adoption of interdiff as a sponsoring format, the following methods were amoung those used for the sponsoring of new upstream packages. These methods are no longer preferred.

Package in REVU

Synopsis:

• Candidate packages were uploaded to REVU as with NEW packages. Reviewers would review during normal REVU activities.

Deficiencies:

• New upstream candidates do not need to meet the same criteria as NEW packages, leading to confusion on the part of both uploaders and reviewers as to whether the package should be uploaded.
• Packages are frequently downloaded entire, and not checked against the upstream source.
• REVU workflow is significantly different than the typical merge workflow.
• Packages already in the archive are sorted to the bottom of the REVU list.

Package in other repository or HTTP accessible location

Synopsis:

• Candidate packages were uploaded to accessible location (personal site, shared site, PPA, etc.). Packagers would request sponsorship via email, IRC, or a URL in a bug.

Deficiencies:

• Packages are frequently downloaded entire, and not checked against the upstream source.
• Weak tracking for LP-external sponsoring
• LP bug links to external sites may not be trusted if the sponsor does not trust the submitter.
• Not clear if such packages should be considered fresh uploads or syncs

Package attached to bug

Synopsis:

• Candidate packages were attached to bugs as separate files. Sponsors would download the portions from the bug, and

Deficiencies:

• Huge storage requirements or manual attachment management.
• Awkward upload / download with one attachment per comment.
• Packages are frequently downloaded entire and not checked against the upstream source

["CategoryMOTUUbuntuPackagingMerge"]