Merging

Redirected from page "MOTU/Packages/Merging"

Clear message

Background & History

Ubuntu is based on the Debian GNU/Linux distribution and uses the same package management system. In the beginning of each Ubuntu development cycle the packages in Ubuntu are updated to those in Debian unstable. However, because Ubuntu is not the same as Debian, some of the packages need to be modified to work in Ubuntu. Sometimes there are also bug fixes that are present only in Ubuntu version of a package. To determine if a package contains Ubuntu specific fixes, take a look at its version. If version includes ubuntu in it (an example would be gimp 2.2.9-3ubuntu2) then the Ubuntu developers have made some change and it is no longer the same as the Debian package. There are more than 1000 such packages in the Universe repository.

Some packages also have a version differing via a build suffix, meaning that the package was reuploaded without any changes to the archive to trigger a rebuild against newer dependencies. Those packages are automatically synced back whenever a new version is uploaded to Debian.

At the start of the development cycle a decision has to be made with regard to these Ubuntu-versioned packages. Of course if the Debian version hasn't changed since the last Ubuntu release then nothing needs to be changed. However, if there is a newer version of the package in Debian then one of two things should happen. If all of the reasons that the Ubuntu version existed (bug fixes, dependencies, etc.) are fixed in the new Debian package (or if the residual difference is sufficiently trivial it's not worth the maintenance overhead of merging) then we can just take the Debian package directly. This is called a sync. However, if the new Debian version has the same issues that caused the Ubuntu version to be made, then those changes need to be applied to the new Debian version. This is called merging.

Useful tools

MoM is the official merge system, hosted on merges.ubuntu.com.

In the future, the preferred way will be to to use bzr with bzr-builddeb plugin to perform merging. For documentation on how to use bzr-builddeb for developing ubuntu see Merging with bzr and related UDD documentation. You can use it now and help the UDD developers by giving feedback on the process.

To grab all the files needed for the merge from MoM, use the grab-merge script in the ubuntu-dev-tools package.

The Process of Merging/Syncing

Bzr-builddeb can handle smart merging of packages from Debian, see UDD documentation and `Merging with UDD

Alternatively Scott's excellent Merge-O-Matic tool (rather the output from that tool) at http://merges.ubuntu.com/ is used to lay the groundwork for the process. On http://merges.ubuntu.com/universe.html, you'll see a pretty lengthy list of pending merges. Near the leading vertical ("top") of the page, you'll see three links, one for outstanding merges, another for new, and another for updated.

  • 'Outstanding' merges are those that have not been tackled in Ubuntu at all across any release.
  • 'New' merges means that no one has tackled these merges yet in the current development cycle.
  • 'Updated' merges means that newer Debian versions are available for merges that have already been done in the current development cycle.

    (X) The 'Updated' section has been disabled as it was incorrect about a large number of packages. Those merges can now be found in the 'Outstanding' section.

The above lists include all the packages that contain Ubuntu changes from their Debian counterparts at the time of the snapshot. Several packages that have previously been imported from Debian to Ubuntu are automatically synced at specific times in each release cycle. Those automatically "synced" packages are not include in the list of pending merges.

Outline of Algorithm for Tackling Merges

  1. Grab the Merge-O-Matic output. You can do this using grab-merge.sh.

  2. Read the Merge-O-Matic output (either in the extracted source or via REPORT on the website).
  3. Inspect the conflicts in each file listed in the M-O-M output, and follow this subroutine:

    1. Prefer the newer Debian changes when possible. This means we discard Ubuntu changes until we have to use one.
      Warning /!\ When packages split and introduce conflicts/replaces statements, we often cannot safely discard the Ubuntu version number, because the new conflicts statement may refer to the Debian version number, but in Ubuntu it needs to be the Ubuntu number.

    2. If newer Debian changes were made that don't affect the old Ubuntu source package, then extract the newer pristine Debian source package and check if those Debian changes allow the Ubuntu merges to be dropped from the merged source.
    3. If those Debian changes allow the Ubuntu merges to be dropped, then make the modifications in the merged source package.
    4. Continue this step for all files listed in the M-O-M output.
  4. Sanity-check the package. For instance, you don't want hardcoded paths for older software (that you just dropped) sitting in the source.

  5. Change the debian/changelog to note your work. Note which Ubuntu deltas are still valid and why invalid ones were dropped. Change the M-O-M e-mail address and name to yours. Make sure you keep old Ubuntu changelog entries. 

    xxxxx (vvv-1ubuntu1) natty; urgency=low

  * Merge from debian unstable.  Remaining changes:
    + Modified build-dep on libfoo to build on natty.
    + Added .desktop file.
  6. Generate a new merged source package using the merge-buildpackage script.

    • Remember to pass any necessary options like " -rfakeroot -k<your_gpg_key_id> ".

  7. Use debdiff to diff your new package against both the old Ubuntu package and the old Debian package to make sure you have carried forward any needed Ubuntu changes and have them all correctly documented. (if a new upstream version is involved, it may be far more useful to diff the old and new debian directories directly rather than just relying on debdiff.)
  8. Make a note of which Ubuntu changes could be forwarded back to Debian to improve their package (details below)

  9. Sanity-build the source package in pbuilder or sbuild, check the contents of the created debs (using dpkg-deb -c foo.deb), and upload or request sponsorship for your merged source package.

Work flow example

Check for needed merges

Fire up your tool! We will use for this example MoM, therefore open this page:

All merges are sorted by component: main, restricted, universe and multiverse. Lets suppose we will work on a package in universe, therefore lets click on universe, or go directly to http://merges.ubuntu.com/universe.html.

You will see now all outstanding merges, listed in alphabetical order. In the left column, titled package, you will see the source package name, and its associated binary packages. There are two links here:

  • One point to a REPORT, which gives general information about the merge, and possible conflicts detected. It would be very usefull to read it before starting to work on the merge.
  • Another point to the source package page in launchpad. You may check there what the package is about, and check as well for any open bug reports.

In the second column, you will see:

  • The maintainer of the last Ubuntu package (the one listed in the changelog)
  • The uploader of the last Ubuntu package
  • The version of the last Ubuntu package

Before starting to work on the merge, you should check with the previous maintainer and/or the previous uploader, if they intend to work on the merge themselves. They might give you useful tips as well, even if they delegate the task to you. MoM also has a comments field where you can often find notes about the merge.

The last two columns give the Debian version we want to merge from, as well as the previous Debian version (often called the base version).

Get the files you need

The first thing to do is to get the package sources. Since we are using MoM, we will use for this example the grab-merge script from ubuntu-dev-tools.

  • Fetch all the needed files with grab-merge: grab-merge <sourcepackagename> (it will download files into a new subdirectory of the current directory named <sourcepackagename>

  • Once all the files are downloaded, check again REPORT. It contains useful information on the merge, as well as tips on possible conflicts.

Tip for advanced users: If you decide to download the source packages and do the merge manually without the assistance of grab-merge script, be sure to remember that if the new Debian version and current Ubuntu version are the same upstream version than when you extract the source (ex. dpkg-source -x <sourcepackage>.dsc) they'll both try extracting to the same directory.

Check if its a merge or a sync

  • Check the "old ubuntu".patch file to check what has been modified by Ubuntu in previous versions and why.
  • Check the "new debian".patch file to check what changes have been implemented by Debian in this new version.
    • tip: you may want to use filterdiff to filter out all changes external to /debian.
  • Check if the changes made in Ubuntu have been merged in Debian. If it is not the case, try to find out why, and if these changes are still necessary.

  • If all the Ubuntu changes can be dropped, you need to file a sync bug.

File a merge bug

If you are not the previous uploaders, before you work on the merge (if it is before Debian Import Freeze, also check with the last Ubuntu uploader before doing the merge), file a bug in Launchpad against the source-package product:

  • Set the bug title to: Please merge <sourcepackagename> <debian-version> (repository) from Debian <repository> (<component>)

    • For instance: Please merge zoph 0.7.0.2-2 (universe) from Debian unstable (main)

  • Set the bug status to "in-progress" and assign it to yourself.
  • If you have the permissions to do so, set the importance to "Wishlist". Otherwise, there is no need to set this field.
  • Note down the bug number, you will need it later.

Now go back to the m-o-m summary page (for example https://merges.ubuntu.com/universe.html) and leave a comment linking to the bug. This way, others know that there is already a bug filed for that merge:

  • Click in the Comment column on the invisible text entry field
  • Leave a comment like "bug #123456" and press Enter
  • The page will update and link to your bug

Work on the merge

So there are still changes to carry over from Ubuntu to the Debian package source? Look closely at the patches provided and try to make the minimal change to set the package right.

You should check as well any open bug report, maybe your merge will close some of them, or perhaps there is a simple fix that you can include in your merge.

An already patched source tree has been prepared by the tool in the directory you have chosen. You should browse it, check that all the relative Ubuntu changes have been applied, and resolve any conflict. grab-merge.sh could already indicate some conflicts; they are marked as follows in the relative source file(s): 

<<<<<<< packagename-version (ubuntu)
changes from ubuntu here
maybe several lines long
=======
original debian lines
again its possible that there are several lines of them
>>>>>>> packagename-version (debian)

You should manually resolve the conflict choosing either version (or possibly "merging" them together).

Once you're done patching the package, don't forget to update debian/changelog. Use dch to update the file, or modify it manually. Add as many infos as necessary to make your changes fully understandable. Check also that the new package version is set correctly (should be -Xubuntu1, where X is the Debian version on which the merge is done) and that the distribution is correct.

Remember that you should list ALL Ubuntu changes that were carried over. Remember to add an (LP: #xxxxx) metatag where xxxxx is your merge bug report number and remove any previous metatag. Remember to add your name and email too in place of "Ubuntu Merge-o-Matic <mom@ubuntu.com>"!

Build, check and report

After you're done with this, rebuild the package with debuild -S, and check that it builds and install without errors.

If you need sponsorship, you should generate two debdiffs: one between the Debian .dsc and your modified one and another one between the old Ubuntu .dsc and your modified one. Check both debdiffs for anything which is extraneous to your changes. If needed, filter out any extraneous changes, either manually or with filterdiff.

Please check that your debdiff is of an acceptable size before uploading it; this is especially usefull for the debdiff between the old Ubuntu package and your modified one, that will contain all upstream changes too. If the size is too big, just leave a comment to the sponsors that you have it available and can upload it if necessary.

Attach these debdiffs to the bug report, un-assign yourself, set the status to "confirmed" and subscribe (not assign!) the relative sponsor (ubuntu-sponsors team).

Forward patches upstream

If any of the changes applied to the package are useful beyond Ubuntu, they should be forwarded to the relevant Debian package maintainer or to the upstream project maintainer. Packaging changes should be forwarded to Debian, while non-packaging changes may be sent to either maintainer as appropriate. If you notice that this is part of a bigger change that deliberately differs from Debian, consider adding it to UbuntuPackagingChanges. Check the Debian Bug Tracking System (BTS) to see if the change has already been sent to Debian. Debian has far more developers than Ubuntu and generally Debian's package maintainers are experts in their particular packages. Both from a maintainability and quality perspective keeping packages in sync is highly desirable.

Reference: Debian/Bugs

Get your work uploaded

Check from time to time your bug report (you should be warned if changes or additions are made) and work on any comment the sponsors may have. Once accepted, your bug report will be automagically closed by your (LP: #xxxxx) metatag.

When uploading, be sure to check if the merge represents a new upstream version, in which case the changes file must refer to the orig.tar.gz. Also be sure to include all debian changes in your upload. (debuild -S -sa -vX.YubuntuZ).

Thanks for your contribution!

Things to check during the merge

  • Does the package still require the Ubuntu delta?
    Sometimes changes in other packages will cause the Ubuntu delta to no longer apply, and the change can be dropped, even if not included in Debian.

  • Does the new Debian revision close any Ubuntu bugs?

    • Be sure to check the bug page in Launchpad for the package. Also, take a look at the Debian bugs that are closed in the Debian changelog. If bugs are closed, add a section to the changelog detailing the bugfixes, with the appropriate (LP: #bugnumber) entries.
  • Do any of the changes in the remaining Ubuntu delta apply to Debian?
    • If so, make sure that the patch has been submitted to Debian (create a new bug in the BTS if required).

      • If you have a local MTA installed, make use of submittodebian (in the ubuntu-dev-tools package),

      • if not please respect Debian/Usertagging and add the necessary tags to your email.

    • If the issue addressed an Ubuntu bug, make sure that the Ubuntu bug is linked to the relevant Debian bug.

  • Checking the debian bug tracking system at http://bugs.debian.org/src:<packageName> might also prove useful.

Questions

If you are just starting with merges, and have any questions not resolved by the above description, please add them below, and the answers will be merged into the documentation above by the merging page editors. Please be sure to subscribe to the page if waiting for an answer, so as to ensure that you will be notified when the answer is available.

  • Add your question here...
    • Could someone please add some information about the workflow when there are no conflicts? I would look at the *ubuntu1.patch, and work on the changelog, using the previous merges' changelog as a template, sanity checking each of those entries against the current patch. Is this correct?

    • Can someone please explain all the color codes used? They represent priority 'Priority' of the package

    • What if http://merges.ubuntu.com/ does not contain a merge for the newest Ubuntu version? How to create the *ubuntuX.patch manually from the newest source package?

Go back to UbuntuDevelopment.
CategoryProcess
CategoryUbuntuDevelopment

UbuntuDevelopment/Merging (last edited 2022-09-22 14:36:38 by eslerm)