MergingBestPractices

This is the log of a MOTU School session on merging presented by Nathan Handler (nhandler)

[23:01] <nhandler> Hello everyone, and thanks for joining me for my MOTU School session about merging packages from Debian.
[23:01] <nhandler> If you have any questions, please feel free to ask them in #ubuntu-classroom-chat at any time. Please be sure to prefix your questions with "QUESTION:".
[23:02] <nhandler> The wiki does an excellent job of explaining why we need to merge/sync packages from Debian.
[23:02] <nhandler> Here is an excerpt from https://wiki.ubuntu.com/UbuntuDevelopment/Merging.
[23:03] <nhandler> Ubuntu is based on the Debian GNU/Linux distribution and uses the same package management system. In the beginning of each Ubuntu development cycle the packages in Ubuntu are updated to those in Debian unstable. However, because Ubuntu is not the same as Debian, some of the packages need to be modified to work in Ubuntu. There might also be bug fixes that Ubuntu developers have introduced into the packages.
[23:03] <nhandler> You can determine whether this has taken place by noting the package version. If the package version includes ubuntu in it (an example would be gimp 2.2.9-3ubuntu2) then the Ubuntu developers have made some change and it is no longer the same as the Debian package. There are more than 1000 such packages in the Universe repository.
[23:03] <nhandler> At the start of the development cycle a decision has to be made with regard to these Ubuntu-versioned packages. Of course if the Debian version hasn't changed since the last Ubuntu release then nothing needs to be changed. However, if there is a newer version of the package in Debian then one of two things should happen.
[23:04] <nhandler> If all of the reasons that the Ubuntu version existed (bug fixes, dependencies, etc.) are fixed in the new Debian package then we can just take the Debian package directly. This is called a sync. However, if the new Debian version has the same issues that caused the Ubuntu version to be made, then those changes need to be applied to the new Debian version. This is called merging.
[23:04] <nhandler> The Ubuntu Merge-o-Matic (MoM), http://merges.ubuntu.com/universe.html, provides a list of packages that need to either be merged or synced from Debian.
[23:05] <nhandler> DaD, http://dad.dunnewind.net/universe.php, is another merge tool. Although we will be using MoM for our example today, you should always check DaD to make sure that someone isn't already working on merging the package you want to do. You also should ask the last Ubuntu uploader if they are planning on doing the merge.
[23:05] <nhandler> Today, we will be merging a package called "ksensors".
[23:05] <nhandler> Before we begin, please install: devscripts, build-essential, wget, fakeroot, quilt, patchutils, and debhelper from the repositories. You will need them in order to follow along with this lesson.
[23:06] <nhandler> All of these packages are available in the Ubuntu repositories.
[23:06] <nhandler> Start by creating a directory called "Merges" in your home folder. This folder will hold all of your merges.
[23:07] <nhandler> Next, download the grab-merge script, http://merges.ubuntu.com/grab-merge.sh, to the Merges folder, and make it executable with 'chmod +x ~/Merges/grab-merge.sh'.
[23:08] <nhandler> We will use grab-merge.sh to download all of the files that we need to perform the merge from MoM.
[23:08] <nhandler> Inside your Merges folder, create a new folder called "ksensors". This new folder will hold all the files related to merging ksensors.
[23:09] <nhandler> Now, open a terminal (Applications->Accessories->Terminal), and type 'cd ~/Merges/ksensors' to enter our new ksensors folder.
[23:09] <nhandler> We will now download the files we need to perform the merge. Type '../grab-merge.sh ksensors'. This will download all the files we need.
[23:11] <nhandler> Once the script finishes running, type 'less REPORT' to view the report file that MoM generated for us. This report will show us which files had changes that were unable to be merged automatically.
[23:11] <nhandler> Near the bottom of the REPORT file, you should see a line that looks like this:
[23:11] <nhandler> C  debian/control
[23:12] <nhandler> That means that MoM was unable to automatically merge the changes that were made in Ubuntu and Debian for that file.
[23:13] <nhandler> First, 'cd ksensors-0.7.3-16ubuntu1' to enter the source directory.
[23:13] <nhandler> The first thing you should do is look at the changelog file. This file shows all changes that were made in Ubuntu and in Debian. I prefer to just leave this file open so that I can refer to it later.
[23:14] <nhandler> When performing a merge, we need to focus on the most recent "batch" of Ubuntu changes. You will be able to recognize this "batch", because all of the Ubuntu changes in it will be against the same Debian revision. The debian revision is located after the '-' and before the 'ubuntu'. In ksensors, the batch of changes that we want to focus on includes versions 0.7.3-15ubuntu1 and 0.7.3-15ubuntu2. Do you notice that both of those versio
[23:14] <nhandler> ns have a Debian revision of 15?
[23:15] <medo_> nhandler: do you mean the upstream changelog file
[23:15] <medo_> or the /debian/changelog
[23:15] <nhandler> medo_: I mean debian/changelog
[23:16] <medo_> thank
[23:16] <medo_> :)
[23:16] <nhandler> Remember, please try and ask all questions in #ubuntu-classroom-chat
[23:16] <nhandler> Moving on, if you look at version 0.7.3-15ubuntu1, you will see that it had two changes. It modified the menu file, and it added dh_iconcache to debian/rules.
[23:17] <Laney> <m_newton> QUESTIN where are we lookinh
[23:17] <Laney> looking* I think
[23:18] <m_newton> Laney, I found it thanks
[23:18] <nhandler> In 0.7.3-15ubuntu2, you will see that several changes were made. quilt was added as a Build-Depends in debian/control, debian/rules was updated, a patch was added in debian/patches, and dh_iconcache was changed to dh_icons in debian/rules.
[23:20] <nhandler> Now that we know what changes were made in Ubuntu, we need to look at the most recent Debian version, 0.7.3-16, to see if any of the Ubuntu changes were added upstream in Debian. As you can see from the changelog entry, Debian has not applied any of the Ubuntu changes.
[23:20] <Laney> <m_newton> QUESTION: do we close the terminal where we did less REPORT
[23:21] <nhandler> m_newton: You can close the REPORT file if you wish. It does not matter.
[23:21] <Laney>  <Syntux> Laney, why we have to focus on the version before the last (15) and not the last (16)
[23:22] <Alan_M> Guys, please see nhandlers comment about questions.
[23:22] <nhandler> Syntux: The reason we are looking at the Ubuntu changes that had a Debian revision of 15 is because 0.7.3-16 is not currently in Ubuntu yet. As a result, no Ubuntu changes have been made for that Debian revision
[23:23] <nhandler> Did that answer your question Syntux ?
[23:23] <Syntux> sure.
[23:23] <Syntux> Thanks.
[23:23] <nhandler> Alright, now for the fun part. We are now able to perform the merge.
[23:23] <nhandler> Start by opening debian/control. You should see a section that looks like this: http://paste.ubuntu.com/37417/
[23:24] <norsetto> nhandler: I have a question before, who is telling us that we should carry over these changes?
[23:24] <nhandler> norsetto: What changes are you referring to?
[23:24] <norsetto> nhandler: the changes from the previous ubuntu versions
[23:25] <nhandler> norsetto: We don't always carry over the Ubuntu changes from previous versions. It depends if they are still needed and whether or not they have been applied upstream
[23:25] <norsetto> nhandler: right, I think this is a pretty important point
[23:26] <nhandler> Our goal is to get all of the Ubuntu changes applied upstream so that we can automatically sync the package when Debian releases a new version
[23:26] <nhandler> norsetto: I talk about this topic a little later in the lesson
[23:26] <norsetto> nhandler: in this particular case, seems like we have been carrying the wrong change for the last few years
[23:27] <nhandler> norsetto: You are probably right. Most of the changes in ksensors should have been sent upstream. However, I don't think we should simply drop them now.
[23:28] <Laney> (they can be sent upstrem as part of this merge!)
[23:28] <norsetto> nhandler: I think is a good idea always to check the bug report that originated a change, in this case bug 45675
[23:29] <norsetto> nhandler: so, seems like the change did exactly the opposite of what the bug report was about ... but anyhow, this is indeed off-topic, so, lets continue
[23:30] <nhandler> Back to debian/control...
[23:30] <nhandler> Start by opening debian/control. You should see a section that looks like this: http://paste.ubuntu.com/37417/
[23:30] <nhandler> The lines above the '=======' are the Ubuntu changes. The lines below it are the Debian changes. Whenever possible, we try to use the Debian changes instead of the Ubuntu ones, but this is not always the case.
[23:31] <nhandler> The first thing we will look at is the maintainer field. The Maintainer field should be set to 'Ubuntu MOTU Developers <ubuntu-motu@lists.ubuntu.com>' for packages in Universe or Multiverse, and to Ubuntu Core Developers <ubuntu-devel-discuss@lists.ubuntu.com> for packages in main or restricted. You can read more about this here: https://wiki.ubuntu.com/DebianMaintainerField
[23:31] <nhandler> An easy way to figure out what repository a package is in is to use: 'apt-cache madison <package>'
[23:32] <nhandler> The XSBC-Original-Maintainer field is used to store the information of the Debian maintainer.
[23:33] <nhandler> For ksensors, do you notice how the Ubuntu XSBC-Original-Maintainer field and the Debian Maintainer field both contain the same information? This means that the Debian maintainer has not changed since the last Ubuntu version. We can simply delete the Debian Maintainer field line from the control file. If the two fields had been different, we would have had to update the Ubuntu XSBC-Original-Maintainer field to contain the new Debian
[23:33] <nhandler> maintainer's information.
[23:34] <nhandler> Looking back at debian/changelog, you can see that the only change Ubuntu made to the Build-Depends field was adding 'quilt' in version 0.7.3-15ubuntu2. Since Debian has also modified the Build-Depends field, we have to merge the different changes together. In this case, all we need to do is add 'quilt' to the end of the Debian Build-Depends line. After doing that, we can remove the Ubuntu Build-Depends line.
[23:34] <Laney> <m_newton> Question: apt-cache madison <package> <<< what does madison signify??
[23:36] <nhandler> m_newton: "apt-cacheĀ“s madison command attempts to mimic the output format and a subset of the functionality of the Debian archive management tool, madison." that is from the apt-cache man page
[23:36] <nhandler> Continuing with the lesson...
[23:36] <nhandler> Now that we have resolved all of the conflicting lines in debian/control, we can remove the '<<<<<<<', '=======', and '>>>>>>>' lines.
[23:37] <nhandler> At this point, you should have a control file that has been successfully merged. If you would like to verify that your control file is correct, you can upload it to http://paste.ubuntu.com/, and paste the URL in #ubuntu-classroom-chat.
[23:41] <nhandler> You can view my control file here: http://paste.ubuntu.com/37304/
=== greg__ is now known as __greg__
[23:43] <nhandler> MoM attempts to merge as many of the changes as possible from the Debian and Ubuntu versions of the package. However, you should always verify that this task was done correctly.
[23:44] <nhandler> The first change we will verify is the change to the menu file.
[23:45] <nhandler> Start by opening up debian/menu
[23:46] <nhandler> Notice how they have changed the section from Apps/System to Apps/Tools?
[23:46] <nhandler> This is the change norsetto and I were talking about earlier
[23:47] <nhandler> However, for the purposes of this lesson, just notice how the change made in Ubuntu has not been applied upstream in Debian. It is still needed, and MoM has successfully applied the change.
[23:49] <nhandler> I'm surprised nobody asked me how I knew it changed from Apps/System to Apps/Tools
[23:49] <m_newton> ya, how nhandler
[23:49] <m_newton> I just see Apps/Tools
[23:50] <nhandler> I knew this by opening up ~/Merges/ksensors/ksensors_0.7.3-15ubuntu2.patch
[23:50] <nhandler> This file is a diff that shows all of the Ubuntu changes that were made
[23:50] <nhandler> There is a similar file, ksensors_0.7.3-16.patch, that shows the changes that were made in the new Debian version of the package
[23:51] <nhandler> Now, I want you to look back at the changelog. Notice how in version 0.7.3-15ubuntu1 they added dh_iconcache to debian/rules, and in version 0.7.3-15ubuntu2 they replaced dh_iconcache with dh_icons?
[23:52] <medo_> yeah
[23:52] <nhandler> This means that our current debian/rules should have dh_icons, and not dh_iconcache. We can verify this by looking at debian/rules. You will notice that on line 99 is dh_icons.
[23:53] <nhandler> While we are looking at debian/rules, we should also verify that it includes 'patch' and 'upatch', which were added in version 0.7.3-15ubuntu2. You will find 'patch' on line 30, and 'unpatch' on line 45.
[23:54] <nhandler> While I've been presenting this lesson, james_w was kind enough to find a way to hopefully enable us to build this package.
[23:55] <nhandler> To apply this fix, locate this line "cp -f /usr/share/aclocal/libtool.m4 admin/libtool.m4.in". It should be line 33
[23:56] <nhandler> Right below it, paste these lines of text: http://paste.ubuntu.com/37565/
[23:56] <nhandler> Once you make those changes, you can save and quit the debian/rules file
[23:57] <nhandler> We are almost done. The only other Ubuntu change that we need to verify is still present is 01_kubuntu_fix_etc_sensors_conf.diff.
[23:59] <nhandler> You will find that patch in debian/patches. It is also listed in debian/patches/series.
[23:59] <nhandler> However, since the file that the patch modifies, src/lmsensors.cpp, has been changed in Debian, this patch will fail to apply if we were to try and build the package.
[23:59] <nhandler> Since working with patches is outside the scope of this lesson, I have made the needed changes to the patch. You can find my revised version here: http://paste.ubuntu.com/37450/
[00:00] <nhandler> Please modify your copy of 01_kubuntu_fix_etc_sensors_conf.diff to match my patch.
[00:01] <Laney> QUESTION: Debian have bumped to libsensors4. Shouldn't we investigate this patch more closely as it looks tied to libsensors3?
[00:01] <nhandler> Laney: That is a very good point.
[00:02] <nhandler> If you look at the changelog, you will see that Richard Johnson (nixternal) was the one who made this change
[00:03] <nhandler> If you were doing this merge for real, you would want to talk to Richard as well as look at Launchpad Bug #194103 to verify that this change is still needed.
[00:03] <ubot5> Launchpad bug 194103 in ksensors "ksensors is not finding k8sensors so it is no longer reporting cpu core temps" [Medium,Fix released] https://launchpad.net/bugs/194103
[00:03] <nhandler> I have already taken care of this, and the change is still needed.
[00:03] <nhandler> However, we need to modify the patch slightly to allow it to be applied to the modified file
[00:04] <nhandler> Did that answer your question?
[00:04] <Laney> I trust you ;)
[00:04] <Laney> (yes)
[00:04] <nhandler> Now that we know all of the changes we want are present, we need to document these changes in debian/changelog.
[00:04] <nhandler> Enter 'dch -e' to edit the changelog. You should see an entry that resembles this: http://paste.ubuntu.com/37315/
[00:05] <nhandler> If anyone gets any errors when they run that command, it is most likely because you do not have devscripts installed.
[00:06] <nhandler> There are a few things you should notice about the changelog entry. First, on the first line, it says 'intrepid'. This is because Ubuntu 8.10, Intrepid Ibex, is the current development release of Ubuntu. Second, the last line of the entry should have your name and email address.
[00:08] <nhandler> Before we can finish this changelog entry, we need to create a bug report on https://launchpad.net/. Somebody has already done this, and you can view the report here: https://bugs.launchpad.net/ubuntu/+source/ksensors/+bug/254660. Please refrain from uploading the patches you will produce in a little bit to this bug report. I will take care it so that we don't have everyone in this channel uploading their patch. However, you do want
[00:08] <nhandler> to copy the bug number, 254660, from the report.
[00:08] <ubot5> Launchpad bug 254660 in ksensors "Please merge ksensors-0.7.3-16(universe) from debian unstable" [Wishlist,Confirmed]
[00:08] <nhandler> Now, go back to the changelog, and edit the line that says "* Merge from debian unstable, remaining changes:" to say "* Merge from debian unstable, remaining changes (LP: #254660):". The '(LP: #254660)' part will cause the bug to be marked as 'Fix Released' once the patch has been uploaded.
[00:09] <nhandler> We now need to document what Ubuntu changes we kept. These are the Ubuntu changes that have not been added upstream in Debian, but are still needed.
[00:10] <nhandler> We now need to document what Ubuntu changes we kept. These are the Ubuntu changes that have not been added upstream in Debian, but are still needed.
[00:10] <nhandler> The Ubuntu changes that you need to include are:
[00:10] <nhandler> Add quilt as a Build-Depends for patch management
[00:10] <nhandler> Modify debian/menu to install in kmenu/utilities instead of kmenu/system
[00:10] <nhandler> Add 01_kubuntu_fix_etc_sensors_conf.diff to have ksensors use /etc/sensors3.conf instead of sensors.conf
[00:10] <nhandler> Add dh_icons to debian/rules
[00:10] <nhandler> Update debian/rules for patch and unpatch
[00:13] <nhandler> Aside from the Ubuntu changes that we kept, we also introduced a brand new one.
[00:13] <nhandler> This change was the one from james_w that we added in debian/rules to allow us to build the package
[00:14] <nhandler> We need to make sure we add something like "admin/config.{sub,guess} are removed by libtool, copy them back to allow this package to be built again."
[00:14] <nhandler> Once you make these changes, please upload you changelog file to http://paste.ubuntu.com/, and paste a link in #ubuntu-classroom-chat.
[00:16] <nhandler> You can view my debian/changelog entry here: http://paste.ubuntu.com/37569/
[00:16] <nhandler> The only thing left to do now is generate the patch. First, we have to build the source package. To do this, type 'debuild -S'. If you don't have a GPG key, you can do 'debuild -S -us -uc'.
[00:21] <nhandler> This will create a file called ksensors_0.7.3-16ubuntu1.dsc. We now need to make sure that our new version of the package builds in a clean intrepid environment.
[00:22] <nhandler> I usually use pbuilder to do this. You can read https://wiki.ubuntu.com/PbuilderHowto to learn about creating an intrepid pbuilder chroot.
[00:22] <nhandler> If you already have an intrepid pbuilder chroot created, you can run this command to try and build our new version of ksensors:
[00:22] <nhandler> sudo pbuilder build ~/Merges/ksensors/ksensors_0.7.3-16ubuntu1.dsc
[00:25] <nhandler> Hopefully, james_w's fix should allow pbuilder to build the package without any errors
[00:26]  * james_w crosses his fingers
[00:26] <nhandler> If it builds successfully, you should end up with a .deb file for ksensors in /var/cache/pbuilder/result
[00:27] <nhandler> Once it finishes building, we have to generate 2 debdiff files. One will show the changes between the Debian version, 0.7.3-16, and this one, 0.7.3-16ubuntu1. The second one will show the changes between the last Ubuntu version, 0.7.3-15ubuntu2, and this one, 0.7.3-16ubuntu1. You can generate the Debian->Ubuntu debdiff by running this command from ~/Merges/ksensors:
[00:29] <nhandler> Well, it looks like the fix still didn't work. We are most likely missing a Build-Depends in debian/control. pbuilder produces this error "/bin/sh: ifneq: not found"
[00:30] <nhandler> We won't worry about this, and we will assume that it built successfully.
[00:30] <nhandler> The command to generate the Debian->Ubuntu debdiff is:
[00:30] <nhandler> debdiff ksensors_0.7.3-16.dsc ksensors_0.7.3-16ubuntu1.dsc > ksensors_0.7.3-16ubuntu1.debdiff
[00:31] <nhandler> This is where many people accidentally make mistakes when performing a merge. They simply attach the debdiff to the bug report without even looking at it. You should ALWAYS open up the debdiff, and verify that it only contains the changes you want. For the Debian->Ubuntu debdiff, all of the changes you see should be listed in debian/changelog.
[00:32] <james_w> nhandler: the "ifneq" line shouldn't be indented
[00:34] <nhandler> james_w: It must have gotten indented when I was copying and pasting. I'm rebuilding it now
[00:34] <nhandler> Back to the lesson, after verifying that the debdiff had the correct changes, you would then attach that patch to the bug report that we made earlier.
[00:35] <nhandler> You can generate the Ubuntu->Ubuntu debdiff by running this command from ~/Merges/ksensors:
[00:35] <nhandler> debdiff ksensors_0.7.3-15ubuntu2.dsc ksensors_0.7.3-16ubuntu1.dsc > ksensors_0.7.3-16ubuntu1.debdiff
[00:35] <nhandler> You might be wondering why we have to generate an Ubuntu->Ubuntu debdiff. The reason for this is because in some instances, the Ubuntu->Ubuntu debdiff is smaller than the Debian->Ubuntu debdiff. This makes it easier to check for mistakes, and then sponsor. This usually happens when the most recent Ubuntu batch of changes was very large, and the most recent Debian version introduced very little changes.
[00:37] <nhandler> After verifying that the Ubuntu->Ubuntu debdff only contains the changes you want, you would then attach that patch to the bug report as well.
[00:37] <nhandler> This step isn't required, but it is still a good thing to do. You should add a comment in the bug report on Launchpad, that contains the debian/changelog entry that we made. This allows a person who is reviewing the merge to quickly figure out what Ubuntu changes were kept in the merge.
[00:38] <nhandler> The last step is to add our bug to the sponsorship queue (https://wiki.ubuntu.com/MOTU/Sponsorship/SponsorsQueue). Once it is in the queue, a MOTU will review our patch. If it is acceptable, he/she will upload it.
[00:39] <nhandler> To add our bug to the queue, set the Status to 'Confirmed', and make sure it isn't assigned to anybody. Now, we need to subscribe, not assign, 'ubuntu-universe-sponsors' to the bug since the package is in Universe. If it had been in main or restricted, we would subscribe 'ubuntu-main-sponsors' instead.
[00:39] <nhandler> After you have done that, I would suggest adding a comment on DaD that says 'Bug #254660'. This makes it easier for other people to quickly figure out what packages still need to be merged/synced.
[00:40] <nhandler> Now, you are done with the merge!
[00:41] <nhandler> At this point, I would like to apologize. This lesson did not go nearly as smooth as I originally intended.
[00:41] <nhandler> I still hope that at least some of you learned something from listening to it.
[00:42] <medo_> thanks a lot it was really helpfull and enjoyable
[00:42] <nhandler> I gave a similar lecture a while ago. You can find the IRC log on my wiki. Feel free to read through it if you are interested. https://wiki.ubuntu.com/Cheater7/Lectures/Merges_080613.
[00:43] <nhandler> Does anyone have any questions?
[00:44] <medo_> Nop
[00:44] <nhandler> If you have any interest at all in merging packages. Please feel free to ask me for help.
[00:46] <nhandler> Well, if nobody has any more questions, I guess the lesson on merging packages from Debian is over.

MOTU/School/MergingBestPractices (last edited 2008-08-15 14:46:38 by 92-236-87-214)