Merging-and-Syncing

Attachment 'crimsun-motu-school-trimmed.txt'

Download

   1 23:05:48  <crimsun> ok, let's go ahead and get started
   2 23:06:14  <crimsun> Background stuff first:
   3 23:06:51  <crimsun> Tonight's section will address a couple ways to handle merging Ubuntu changes into newer Debian packages.
   4 23:07:24  <crimsun> Just by way of setting up, it will be useful to glance at http://merges.ubuntu.com/ . You'll also find it useful to go ahead and create a scratch directory (like /tmp/merges/) and install the 'build-essential', 'devscripts', 'cdbs', 'dpatch', and 'fakeroot' packages using apt-get/aptitude/dselect/synaptic/adept. Also consider grabbing the grab-merge.sh script that's mentioned on the merges Web site.
   5 23:09:03  <crimsun> (I'll give everyone a couple minutes while I cover some background.)
   6 23:11:55  <crimsun> It's useful to look at “merging” historically. When the first group of MOTU started back in the Hoary dev cycle, we were pushing a lot of cool (saner) packaging changes (spec'd by the core-dev team) into universe and multiverse. Many of the merges today still bear that legacy, given the sheer number of packages (over 18,000) in Debian. So from Hoary through Dapper, the major “transitions” we've propagated to universe an
   7 23:13:05  <crimsun> ...the major “transitions” we've propagated to universe and multiverse include:
   8 23:13:23  <crimsun> * 1)Modular X.Org (including /usr/X11R6/ -> /usr/ and GL{,u})
   9 23:13:32  <crimsun> * 2)Python 2.4 by default
  10 23:13:41  <crimsun> * 3)GCC 4.0 by default, C++ ABI bump (c2)
  11 23:13:50  <crimsun> * 4)C++ standard allocator change (c2a)
  12 23:13:58  <crimsun> * 5)Numerous library renames (libaa1, libmysqlclient15off, etc.)
  13 23:14:11  <crimsun> * 6)iconcache for faster startup
  14 23:14:26  <crimsun> For Edgy, the major ones we're dealing with are:
  15 23:14:35  <crimsun> * 1)GCC 4.1 by default
  16 23:14:51  <crimsun> * 2)X.Org 7.1 (after resyncing modular X.Org with Debian – this ties into the modular X.Org transition)
  17 23:15:02  <crimsun> * 3)Updated Python policy (python-central, python-support)
  18 23:15:20  <crimsun> The above nine transitions cover nearly every type of Ubuntu merge you'll encounter. The lesser ones (including .desktop files) are probably the most straightforward merges to do.
  19 23:15:58  <crimsun> Personally, the best way to physically “handle” a merge is to do one. We're currently using Scott's excellent Merge-O-Matic tool (rather we're using the output from that tool) at http://merges.ubuntu.com/ .
  20 23:17:08  <crimsun> So, presuming everyone has a scratch directory created and the grab-merge.sh script downloaded, let's dive in.
  21 23:18:48  <crimsun> On http://merges.ubuntu.com/universe.html, you'll see a pretty lengthy list. We'll pick a few of these packages and walk through merges.
  22 23:19:48  <crimsun> Near the leading vertical ("top") of the page, you'll see three links, one for outstanding merges, another for new, and another for updated.
  23 23:21:24  <crimsun> We have no outstanding merges, which is a relief. On the other hand, we have 269 'new' merges, meaning that no one has tackled these merges yet in the Edgy dev cycle. There are, though, 47 'updated' merges, meaning newer Debian versions are available for merges that have already been done in the Edgy dev cycle.
  24 23:21:54  <Fujitsu> What's the difference between New and Outstanding?
  25 23:22:42  <Fujitsu> (and the updated merges thing can't always be trusted, like in the case of carpaltunnel)
  26 23:23:02  <crimsun> Outstanding ones have not been tackled in Ubuntu at all across any release.
  27 23:23:26  <carthik_home> Also, do the lists include all the packages in Debian?
  28 23:24:00  <crimsun> The lists include all the packages that contain Ubuntu changes from their Debian counterparts at the time of the snapshot.
  29 23:24:02  <Amaranth> carthik_home: All the ones that need attention. If the Ubuntu package didn't change it is synced automatically.
  30 23:24:18  <carthik_home> I mean aren't there any packages in Debian that are not in Ubuntu yet? ( I've heard it said that the deb repo is more vast than Ubuntu's)
  31 23:24:46  <Amaranth> carthik_home: They are autosynced.
  32 23:24:49  <carthik_home> (sorry if I am digressing - let me know and I'll can it)
  33 23:25:12  <crimsun> Because the snapshots are discrete, there may well be a few new Debian packages that aren't on the list, but they will be synced from Debian at a future time.
  34 23:26:06  <carthik_home> okay
  35 23:26:24  <crimsun> For completeness, Lucas (another MOTU) maintains a merge list, too, at http://tiber.tauware.de/~lucas/mergescountdown/
  36 23:27:04  <crimsun> (He uses different tools and a different update procedure.)
  37 23:27:43  <crimsun> So let's go ahead and dig in.
  38 23:29:20  <crimsun> Let's take a look at a few that are listed for me, starting with 'xmakemol' at http://merges.ubuntu.com/x/xmakemol/REPORT .
  39 23:30:01  <crimsun> For reference, I have a /tmp/merges/ directory and /tmp/grab-merge.sh
  40 23:30:37  <crimsun> From within /tmp/merges/, I'll execute: ../grab-merge.sh xmakemol
  41 23:31:29  <crimsun> The tool downloads the Debian and Ubuntu source packages and places them in your current directory, overwriting previous content.
  42 23:31:54  <crimsun> If there are special changes that you need to look at, they'll be listed.
  43 23:32:11  <crimsun> This output matches what is described in the REPORT.
  44 23:34:03  <crimsun> Note that there appears to be a conflict in the Ubuntu and Debian changes in (on my system) /tmp/merges/xmakemol-5.15-1ubuntu1/debian/control
  45 23:34:28  <crimsun> Now let's look at the actual conflict by opening that file in your favourite editor.
  46 23:37:09  <crimsun> To understand this conflict, I need to give a bit of background on the very first set of changes described by historical (1).
  47 23:38:19  <crimsun> When Ubuntu went the modular X.Org route, many of the source packages that had formerly been built using 'xlibs-dev' had to be adjusted to work with the new Ubuntu package names.
  48 23:39:21  <crimsun> Some of these changes were straightforward, like replacing 'xlibs-dev' with 'libx11-dev' and any additional ones (libxext-dev, libxinerama-dev, and so on).
  49 23:40:40  <crimsun> The GL packages were a particularly interesting transition. Instead of using 'xlibmesa-gl-dev' (which no longer existed in Ubuntu), we had to use 'libgl1-mesa-dev | libgl1-dev'.
  50 23:41:52  <crimsun> The use of the pipe ('|') as an alternate was to allow the packages to also be buildable in Debian. A similar thing occurred for GLU: We had to use 'libglu1-mesa-dev | libglu1-dev' instead of 'xlibmesa-glu-dev'.
  51 23:43:43  <crimsun> Now here's where the story gets interesting. Because we're addressing merges in Edgy, we need to know what Edgy currently provides. It turns out that in resyncing our modular X.Org packages with Debian's, we've taken up many of Debian's package names, which means that 'xlibmesa-gl-dev' exists once again. On the other hand, 'xlibmesa-glu-dev' still does not exist.
  52 23:44:51  <crimsun> It's never too late to point out that the ultimate goal of "merging" is to realign Ubuntu's universe packages with their Debian counterparts.
  53 23:45:47  <crimsun> Thus, we should always strive to drop Ubuntu changes and go with the newer Debian version (which is called "syncing").
  54 23:46:32  <crimsun> Syncing is important because it removes the maintenance burden from Ubuntu.
  55 23:47:44  <crimsun> Returning to our debian/control example for xmakemol, please notice that the only conflict lies in the Build-Depends field, or what the source package needs installed to generate the binary packages (debs).
  56 23:48:33  <crimsun> Specifically, the Ubuntu change is the use of 'libgl1-mesa-dev | xlibmesa-gl-dev' in place of Debian's 'libgl1-mesa-swx11-dev'.
  57 23:50:30  <crimsun> To understand this specific change, I have to explain that in earlier Ubuntu releases, having libgl1-mesa-swx11 installed forced the removal of accelerated libgl1-mesa, so I opted to go the common route of not forcing that removal.
  58 23:51:24  <crimsun> However, we're now dealing with Edgy having resynced/merged X.Org from Debian, so we should strive to drop Ubuntu-specific changes for fewer maintenance headaches.
  59 23:52:20  <crimsun> There are two possible outcomes when working with a Ubuntu merge listed on the Merge-O-Matic pages: 1) You merge, or 2) You request a sync.
  60 23:53:23  <crimsun> Keeping in mind that we only want to keep Ubuntu-specific changes that are absolutely necessary to the source package's ability to be built, we will decide to drop the Ubuntu merge and request a sync for 'xmakemol' from Debian Sid.
  61 23:54:36  <crimsun> To do that, let's look at the procedure for requesting a sync from Debian Sid.
  62 23:55:03  <crimsun> First, point your Web browser at https://launchpad.net/distros/ubuntu/+search . In the text entry field, type the name of the source package, which is xmakemol .
  63 23:56:31  <crimsun> Second, click the link for the 'Source package "xmakemol"...'
  64 23:56:56  <crimsun> Third, in the upper left corner, there's a "Report a Bug" link that should be clicked
  65 23:58:01  <crimsun> Fourth, we just need to fill in the necessary information. In the Summary text entry field, we could use "Please sync xmakemol from Debian Sid"
  66 23:59:32  <crimsun> In the 'Further information...' field, you will want to state that it's "Ok to override Ubuntu changes"
  67 00:01:40  <crimsun> If you don't have privileges to upload source packages to the Ubuntu processing infrastructure, you'll then need to find a MOTU to approve your sync request, and the MOTU will double-check it and perform any additional steps to subscribe the archive administrators to the bug report. The archive administrators actually run the scripts that sync the source packages from Debian Sid.
  68 00:02:54  <crimsun> A pending sync request looks like: https://launchpad.net/distros/ubuntu/+source/xmakemol/+bug/54405
  69 00:04:21  <crimsun> A completed sync request looks like:
  70 00:04:22  <crimsun> https://launchpad.net/distros/ubuntu/+source/diacanvas2/+bug/53994
  71 00:05:56  <crimsun> That's the process for requesting a sync, which is completed by the archive admins. Now let's look at something that actually requires a merge.
  72 00:07:45  <crimsun> My ready example is unfortunately a bit large, so it make take a while to download. Let's look at wesnoth.
  73 00:07:52  <crimsun> may take, rather
  74 00:09:09  <crimsun> While that's going, we can go ahead and get a good idea of what to consider by reading the current Ubuntu package's changelog and dsc files and comparing them to Debian's dsc.
  75 00:09:28  <carthik_home> so cd /tmp/merges and ../grab-merge.sh wesnoth then?
  76 00:09:39  <crimsun> yep
  77 00:10:00  <carthik_home> you mean http://merges.ubuntu.com/w/wesnoth/REPORT
  78 00:11:02  <crimsun> At the bottom of http://packages.ubuntu.com/src:wesnoth , there's a link to the Edgy source package, which we'll click.
  79 00:11:53  <carthik_home> to get to http://packages.ubuntu.com/edgy/source/wesnoth ?
  80 00:11:54  <crimsun> That brings us to http://packages.ubuntu.com/edgy/source/wesnoth , and at the bottom of this page we can view the current Ubuntu changelog in addition to the dsc files.
  81 00:13:08  <carthik_home> http://changelogs.ubuntu.com/changelogs/pool/universe/w/wesnoth/wesnoth_1.1.7-1ubuntu1/changelog
  82 00:13:14  <crimsun> eoghan: In the 'More Information..' section, click the "View the Debian changelog" link.
  83 00:13:46  <crimsun> The dsc is listed first in the File table in the 'Download wesnoth' section
  84 00:15:01  <crimsun> My normal practice -- and this is of course simply my own approach -- is to have one Web browser tab open for http://packages.ubuntu.com/edgy/source/wesnoth and another open for http://packages.qa.debian.org/wesnoth
  85 00:16:59  <crimsun> The reason for looking at the dscs is to look at the build dependencies; the reason for looking at the changelogs is to see whether Ubuntu changes have been "absorbed" upstream [in Debian] and noted there.
  86 00:19:18  <carthik_home> but the ubuntu changelog doesn't show the latest debian changes -- or does it?
  87 00:19:41  <crimsun> carthik_home: no, it doesn't, but it /should/ document why Ubuntu's diverged from Debian's.
  88 00:19:57  <carthik_home> okay
  89 00:19:59  <crimsun> (the reason for there needing to be a merge in the first place)
  90 00:21:04  <crimsun> The changelog notes that there's one difference, which is installing one file so that a particular use case (the -t switch) succeeds.
  91 00:21:56  <crimsun> We need to check if that file (scenario-test.cfg) is installed according to the Debian source package, and if so, we can request a sync, otherwise we have to merge the Debian changes.
  92 00:22:59  <crimsun> If you've completed downloading the wesnoth merge set, you'll note there are a TON of conflicts
  93 00:24:01  <carthik_home> yes pngs
  94 00:24:17  <crimsun> The good news is that according to the newest Debian changelog (in wesnoth_1.1.8-1.diff.gz and http://packages.qa.debian.org/w/wesnoth/news/20060726T080342Z.html), those are simply location changes.
  95 00:25:04  <crimsun> Thus, we only have to check if scenario-test.cfg is installed according to Debian's packaging structure.
  96 00:25:23  <crimsun> to do that, we'll extract the Debian source package for 1.1.8-1.
  97 00:25:42  <crimsun> From /tmp/merges/ , ``dpkg-source -x wesnoth_1.1.8-1.dsc''
  98 00:26:06  <carthik_home> (err, crimsun - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=337834 says that the test .cfg file was included in ver 1.1-2)
  99 00:27:38  <crimsun> carthik_home: here's where we have to be careful
 100 00:28:35  <crimsun> even though the bug report is closed, if we check the source package, scenario-test actually isn't installed
 101 00:28:46  <crimsun> meaning:
 102 00:28:57  <carthik> and the debian version numbers are confusing too...
 103 00:29:14  <crimsun> Let's first check if the file exists in the source package:
 104 00:29:18  <crimsun> $ find . -name '*scenario-test*'
 105 00:29:19  <carthik> yes
 106 00:29:26  <crimsun> ./data/scenario-test.cfg
 107 00:30:19  <crimsun> now let's check the debian/ packaging infrastructure to see if it's referenced
 108 00:30:46  <crimsun> $ grep -nHr 'scenario-test' debian/*
 109 00:31:48  <crimsun> if the change mentioned in the Debian #337834 is present, it should be in one of debian/*.install
 110 00:31:49  <carthik> no such file or dir
 111 00:32:08  <carthik> sorry found it
 112 00:32:29  <crimsun> [ $(pwd) == /tmp/merges/wesnoth-1.1.8 ]
 113 00:32:37  <carthik> yep, sorry
 114 00:33:03  <carthik> okay.
 115 00:33:28  <crimsun> It turns out that despite the Debian bug being "closed", it's actually still "open" -- the issue isn't resolved, so we need to keep the Ubuntu changes.
 116 00:35:01  <crimsun> sorry, I need to read a bit more closely. :)
 117 00:35:15  <carthik> debian/wesnoth-data.install:11:debian/tmp/usr/share/games/wesnoth/data/scenario-test.cfg
 118 00:35:44  <crimsun> actually it's 'debian/tmp/usr/share/games/wesnoth/data/*.cfg'
 119 00:35:59  <crimsun> which is referenced on line 9 of debian/wesnoth-data.install
 120 00:36:29  <carthik> I'm lost...
 121 00:36:45  <crimsun> make sure you're looking at 1.1.8-1's debian/wesnoth-data.install.
 122 00:36:57  <carthik> the grep gave me that one line I pasted above
 123 00:38:01  <crimsun> I think you're looking in the extracted merged directory.
 124 00:38:15  <crimsun> wesnoth-1.1.8-1ubuntu1/debian/wesnoth-data.install:11:debian/tmp/usr/share/games/wesnoth/data/scenario-test.cfg
 125 00:38:18  <carthik> crimsun, ah yes.
 126 00:38:58  <crimsun> Ok, so it turns out we can avoid merging wesnoth and request a sync for it, hooray.
 127 00:39:01  <carthik> so I had to download the debian sources too?
 128 00:39:21  <crimsun> you already have the Debian source package by using grab-merge.sh .
 129 00:39:41  <carthik> So I had to untar it then?
 130 00:39:58  <carthik> sorry - I missed that step.
 131 00:40:17  <crimsun> right, that was described above.
 132 00:40:50  <carthik> sorry, my bad :)
 133 00:41:02  <crimsun> Just for the record, wesnoth can be synced because *.cfg files are installed by using debian/wesnoth-data.install .
 134 00:41:19  <crimsun> (I was a bit hasty in declaring the bug still open.)
 135 00:41:55  <crimsun> Two syncs so far, which is great for Ubuntu maintenance.
 136 00:43:25  <carthik> crimsun, I am curious - how did you find out that the *.cfg line installs the screnario-test.cfg (since the grep produced zip)
 137 00:44:36  <crimsun> carthik: Ok, I'll need to delve into dh_install(1) a bit, which is a debhelper script to install files into certain packages.
 138 00:44:59  <crimsun> each debian/*.install lists files that are to be installed into that binary package.
 139 00:45:19  <carthik> okay
 140 00:45:47  <crimsun> The original Ubuntu delta comes from scenario-test.cfg not being installed into /usr/share/games/wesnoth/data/
 141 00:46:22  <carthik> ok
 142 00:46:33  <crimsun> So that line [debian/tmp/usr/share/games/wesnoth/data/scenario-test.cfg] was added to debian/wesnoth-data.install
 143 00:47:08  <carthik> okay
 144 00:47:28  <crimsun> In closing Debian bug #377834, the Debian maintainer decided to install *.cfg , hence the [debian/tmp/usr/share/games/wesnoth/data/*.cfg] line in debian/wesnoth-data.install
 145 00:47:58  <crimsun> (since *.cfg covers scenario-test.cfg)
 146 00:48:09  <carthik> yes
 147 00:48:27  <crimsun> Now for a merge :)
 148 00:48:30  <carthik> so this could have been synced earlier, then?
 149 00:48:35  <crimsun> yes
 150 00:48:37  <carthik> okay
 151 00:49:08  <crimsun> Now let's look at m2crypto
 152 00:49:33  <crimsun> This one is a merge because of the Python 2.4 transition noted at the beginning of the session.
 153 00:50:36  <crimsun> (Presuming we've executed grab-merge m2crypto, let's look at the extracted files that have conflicts.
 154 00:50:39  <crimsun> )
 155 00:51:28  <carthik> yes
 156 00:51:33  <carthik> control and rules
 157 00:51:53  <crimsun> First up is debian/control, and the first fields are Build-Depends and Standards-Version
 158 00:52:21  <carthik> so should I extract the debian files from the dsc ?
 159 00:52:36  <crimsun> nope, just use the extracted directory that grab-merge.sh creates.
 160 00:53:15  <carthik> crimsun, okay, so how do I look at the files?
 161 00:53:34  <carthik> I am in /tmp/merges
 162 00:54:08  <crimsun> The grab-merge.sh script will extract a merge candidate for you, which is m2crypto-0.16-1ubuntu1 in this example.
 163 00:54:13  <carthik> /tmp/merges/m2crypto-0.16-1ubuntu1/debian -- okay I am here - which is right, I suppose
 164 00:54:25  <crimsun> Yes.
 165 00:55:14  <crimsun> Then you can use your favourite editor (gedit, kate, vim, emacs, joe, ...) to view/edit the files listed in the output (or REPORT) with a 'C' or 'C*'
 166 00:55:19  <carthik> I can see build-depends and standards-version fields now
 167 00:55:52  <carthik> yes
 168 00:56:03  <crimsun> Our objective, as always, is to use Debian changes and only merge in Ubuntu's if necessary.
 169 00:56:25  <crimsun> To that end, we will drop Ubuntu's Standards-Version and use Debian's.
 170 00:56:43  <crimsun> Thus, we'll delete 'Standards-Version: 3.6.1' and replace it with 'Standards-Version: 3.7.2'.
 171 00:57:29  <crimsun> I'll explain the debhelper versioned dependency now.
 172 00:57:40  <carthik> edit /tmp/merges/m2crypto-0.16-1ubuntu1/debian/control?
 173 00:57:47  <crimsun> carthik: yes.
 174 00:58:42  <carthik> okay I edited line 7 to change 3.6.1 to 3.7.2 - carry on crimsun
 175 00:59:03  <crimsun> (Also make sure that there's only one Standards-Version line.)
 176 00:59:33  <carthik> so do I delete the line altogether, line 7, that is?
 177 00:59:55  <crimsun> sure, that's precisely what I did, for example.
 178 01:00:20  <carthik> okay
 179 01:00:26  <crimsun> Now let's look at the difference in debhelper versioned dependency.
 180 01:00:57  <carthik> carry on
 181 01:01:00  <carthik> ls
 182 01:02:02  <carthik> crimsun, how do I do that?
 183 01:02:25  <crimsun> When the 'Python 2.4 by default' transition was occurring, the ability to have debhelper calculate necessary Python packages for us was added via dh_python(1). To use dh_python in debian/rules, debian/control needed to be updated to depend on a newer version of debhelper, 4.2.28
 184 01:03:12  <carthik> okay - I am with you now - forget the earlier question :)
 185 01:03:42  <crimsun> Thus what we'll do to maintain that dh_python usage in debian/rules will be to keep Ubuntu's specific debhelper versioned dependency in the Build-Depends field.
 186 01:03:53  <carthik> okay
 187 01:04:31  <crimsun> So, currently we have:
 188 01:04:33  <crimsun> <<<<<<< m2crypto-0.13-1.1ubuntu1 (ubuntu)
 189 01:04:33  <crimsun> Build-Depends: debhelper (>= 4.2.28), libssl-dev (>= 0.9.7), python-dev (>= 2.4), python-dev (<< 2.5), python (>= 2.4), python (<< 2.5), swig (>= 1.3.21)
 190 01:04:36  <crimsun> =======
 191 01:04:39  <crimsun> Build-Depends: debhelper (>= 4.1.65), libssl-dev (>= 0.9.7), python, python2.3-dev, python2.4-dev, swig (>= 1.3.24)
 192 01:04:42  <crimsun> Standards-Version: 3.7.2
 193 01:04:45  <crimsun> >>>>>>> m2crypto-0.16-1 (debian)
 194 01:04:50  <carthik> yes
 195 01:05:14  <crimsun> Now let's iterate through the remaining Build-Depends on the Ubuntu side, checking that any newer Debian adjustments are added.
 196 01:05:40  <crimsun> libssl-dev matches, so we can leave it.
 197 01:06:01  <carthik> ok
 198 01:06:33  <crimsun> Ubuntu does not have an m2crypto package built for python2.3, so we'll omit those build dependencies.
 199 01:07:09  <crimsun> (i.e., we'll leave the python* references alone)
 200 01:07:19  <carthik> okay
 201 01:07:39  <crimsun> Finally, we'll update Ubuntu's swig for the stricter/newer version listed in Debian's
 202 01:08:08  <carthik> by editing the ubuntu build-depends entry?
 203 01:08:13  <crimsun> yes.
 204 01:08:34  <crimsun> so we'll change (>= 1.3.21) to (>= 1.3.24)
 205 01:08:41  <carthik> okay
 206 01:08:54  <crimsun> Then, we can remove the diff3 markers for this section (hunk).
 207 01:09:03  <crimsun> That leaves us with:
 208 01:09:04  <crimsun> Build-Depends: debhelper (>= 4.2.28), libssl-dev (>= 0.9.7), python-dev (>= 2.4), python-dev (<< 2.5), python (>= 2.4), python (<< 2.5), swig (>= 1.3.24)
 209 01:09:07  <crimsun> Standards-Version: 3.7.2
 210 01:09:32  <crimsun> The next diff3 hunk is:
 211 01:09:33  <carthik> so delete the debian line and the <<<<<<<<, ========= and >>>>>>>> lines?
 212 01:09:35  <crimsun> <<<<<<< m2crypto-0.13-1.1ubuntu1 (ubuntu)
 213 01:09:36  <crimsun> Depends: ${shlibs:Depends}, python (>= 2.4), python (<< 2.5)
 214 01:09:36  <crimsun> =======
 215 01:09:36  <crimsun> Depends: ${shlibs:Depends}, python2.3
 216 01:09:38  <crimsun> >>>>>>> m2crypto-0.16-1 (debian)
 217 01:09:43  <crimsun> carthik: yes
 218 01:10:16  <carthik> carry on
 219 01:11:17  <carthik> I suppose we are to keep the ubuntu version for this and remove the debian version?
 220 01:11:23  <crimsun> We resolve the conflict above by remember that Ubuntu only specified (and in this example, builds) packages for Python 2.4, so we can delete the Debian version.
 221 01:11:30  <crimsun> remembering, rather
 222 01:11:37  <crimsun> carthik: precisely
 223 01:11:53  <carthik> done
 224 01:12:16  <carthik> so now save the file and exit and proceed to the next one with a conflict?
 225 01:12:25  <crimsun> Then we save the modifications we've made and move on, yes.
 226 01:14:07  <carthik> ok, ready with vim rules
 227 01:16:10  <crimsun> In previous Ubuntu releases, m2crypto was patched. To see which (Ubuntu or Debian) hunks to keep, we'll look at the rest of the package.
 228 01:16:49  <crimsun> So: Check if Debian's changes can be taken over Ubuntu's.
 229 01:17:06  <crimsun> Thus we want to know if versions=$(subst -dev,,\
 230 01:17:06  <crimsun>            $(subst python,,\
 231 01:17:06  <crimsun>              $(filter python%-dev,\
 232 01:17:06  <crimsun>                $(shell sed -n '/^Build-Depends/s/,//gp' debian/control))))
 233 01:17:20  <crimsun> will work instead of:
 234 01:17:27  <crimsun> DEBIAN_DIR = $(shell pwd)/debian
 235 01:17:28  <crimsun> patchdir = $(DEBIAN_DIR)/patches
 236 01:17:28  <crimsun> patches = $(shell ls $(patchdir) | sort)
 237 01:17:28  <crimsun> rev_patches = $(shell ls $(patchdir) | sort -r)
 238 01:17:43  <carthik> ok
 239 01:18:08  <crimsun> Our first clue that we can use Debian's hunk is that there are no references to patches.
 240 01:19:39  <carthik> there are no refs to patches where?
 241 01:19:53  <crimsun> In fact, when we inspect the rest of debian/rules, the only references to the string "patch" lie in Ubuntu hunks.
 242 01:20:27  <carthik> yes
 243 01:21:06  <carthik> but what does the presence or absence of patches in ubuntu/debian chunks signify?
 244 01:21:29  <crimsun> To do that, we need to look at the patches in debian/patches/ .
 245 01:21:51  <crimsun> First up is debian/patches/01_setup.py-distutils-2.4.patch
 246 01:22:00  <crimsun> It's a one-liner:
 247 01:22:01  <crimsun> -def swig_sources (self, sources):
 248 01:22:01  <crimsun> +def swig_sources (self, sources, extension):
 249 01:23:01  <carthik> yes
 250 01:23:58  <crimsun> Now we need to look at the affected file and see if the change has been applied.
 251 01:24:28  <crimsun> Upon checking setup.py, we find that the change has not been applied, so we need to keep this patch.
 252 01:25:11  <carthik> yes
 253 01:25:22  <crimsun> I'll need to discuss ramifications of using patch systems in a second, but let's continue to the next patch.
 254 01:25:35  <crimsun> debian/patches/02_httpslib.py-2.3-and-above.patch
 255 01:27:34  <crimsun> Now we check M2Crypto/httpslib.py and find that the changes are unnecessary.
 256 01:28:48  <carthik> those lines are no longer there in the file...
 257 01:29:08  <crimsun> correct, so upstream has changed it to make the patch unnecessary.
 258 01:29:34  <carthik> okay
 259 01:30:06  <carthik> but what if, in the new source files, there are other lines that mention version numbers, and we have to change those to suit us?
 260 01:30:25  <crimsun> Then we have to change them.
 261 01:31:10  <carthik> so we have to search and find the lines first? - I'm sorry to have interrupted - please go on... :)
 262 01:31:53  <crimsun> In the worst possible case of upstream not documenting such a change, yes, we'd have to search the code (which requires some Python familiarity).
 263 01:32:36  <carthik> alright
 264 01:33:03  <crimsun> What saves us from having to do just that in this example is checking CHANGES.
 265 01:33:12  <crimsun> [m2crypto-0.16-1ubuntu1/CHANGES]
 266 01:34:00  <crimsun> In checking CHANGES, we note in the notes for 0.15 that support for "OpenSSL 0.9.8, Python 2.4.1, SWIG 1.3.24" was added
 267 01:35:01  <carthik> yes
 268 01:35:02  <crimsun> That actually obviates the debian/patches/01_setup.py-distutils-2.4.patch , too.
 269 01:35:29  <carthik> okay
 270 01:36:27  <crimsun> You should double-check, of course, that both patches can be dropped, and to do that, you'll back out to /tmp/merges/ , extract the newer Debian source package, and check if debian/patches/ exists.
 271 01:36:28  <carthik> okay
 272 01:37:48  <crimsun> Upon doing that, you'll find that Debian's 0.16-1 has no debian/patches/ , which confirms our hypothesis that the Ubuntu hunks in debian/rules can be dropped (since no patches need be applied).
 273 01:38:30  <crimsun> Thus, you'll return to the merged source package and remove the Ubuntu-specific changes.
 274 01:38:56  <carthik> okay
 275 01:40:10  <crimsun> After those modifications are saved, we'll need to return to debian/control and clean up the Build-Depends list so that it will work with debian/rules.
 276 01:40:19  <carthik> what if Debian's source had some in debian/patches ?
 277 01:40:39  <carthik> had some patches, I mean.
 278 01:41:02  <crimsun> if Debian's extracted source package had contained debian/patches/*, then we'd need to check if they were valid against the Debian source package.
 279 01:41:41  <crimsun> We would have had to check whether /those/ patches were valid against Debian's 0.16-1.
 280 01:42:00  <crimsun> (speaking strictly of debian/patches/* in Debian's 0.16-1)
 281 01:42:13  <carthik> okay
 282 01:42:34  <carthik> so we now go back to debian/rules and edit that?
 283 01:42:35  <crimsun> I'll outline a brief algorithm after this example to clarify things.
 284 01:42:45  <carthik> that would be great
 285 01:43:09  <crimsun> Correct, if you haven't dropped the Ubuntu changes from debian/rules, that's the next step.
 286 01:44:01  <crimsun> Then you'd fix up debian/control's Build-Depends so that the parsing in debian/rules works correctly.
 287 01:45:45  <crimsun> You do that by changing 'python-dev (>= 2.4), python-dev (<< 2.5)' to 'python2.4-dev' and by changing 'python (>= 2.4), python (<< 2.5)' to 'python'.
 288 01:46:43  <crimsun> The first change is straightforward, since it's just collapsing what is already stated.
 289 01:47:00  <crimsun> The second change is straightforward but not as obvious:
 290 01:47:14  <crimsun> To ensure it's correct, we check:
 291 01:47:26  <crimsun> $ apt-cache depends python && apt-cache policy python
 292 01:48:34  <crimsun> That demonstrates that we'll get "a python2.4", which is semantically identical to what we had before.
 293 01:49:47  <carthik> why not python 2.4 in control then, instead of just 'python' ?
 294 01:50:05  <carthik> 'pthon2.4' I mean
 295 01:50:16  <carthik> d'oh, you get the point.... sorry
 296 01:51:07  <crimsun> 'python2.4' is valid, yes
 297 01:51:39  <carthik> so in control, we have:
 298 01:51:46  <carthik> Build-Depends: debhelper (>= 4.2.28), libssl-dev (>= 0.9.7), python2.4-dev, python, swig (>= 1.3.24)
 299 01:51:46  <carthik> Standards-Version: 3.7.2
 300 01:52:03  <crimsun> On the other hand, the real reason why we want 'python' is answered by looking at the snippet from debian/rules which actually does the substitution:
 301 01:53:12  <crimsun> It takes the "python2.X-dev" strings, strips the "python" and "-dev" bits, and leaves "2.4", which is the Python version(s) for which packages are generated.
 302 01:53:41  <crimsun> Thus while both 'python' and 'python2.4' will work, for consistency we should probably use what Debian uses.
 303 01:54:15  <carthik> okay - for consistency with Debian  -- I understand.
 304 01:55:08  <carthik> okay crimsun
 305 01:55:36  <crimsun> there's one more optional change, which is adjusting debian/control:Depends
 306 01:55:57  <carthik> hmm
 307 01:56:18  <carthik> just make that 'python' too?
 308 01:56:29  <crimsun> yes.
 309 01:56:54  <carthik> okay
 310 01:57:32  <crimsun> Now to be correct, we'd want to make that 'python2.4'.
 311 01:57:39  <carthik> okay
 312 01:58:06  <crimsun> either will work in Ubuntu, since we no longer support Warty and have python2.4 as default in the supported releases.
 313 01:58:35  <carthik> since we are changing from Debian's version, we might as well make it as good as it can be?
 314 01:58:48  <crimsun> Right, that's my approach.
 315 01:59:24  <carthik> okay
 316 01:59:25  <crimsun> After we've modified debian/{control,rules}, we need to check if there are any hardcoded python2.3 paths in debian/*
 317 01:59:59  <crimsun> That can be as straightforward as: grep -nH 'python2.3' debian/*
 318 02:00:43  <carthik> i see one line from a changelog which is immaterial?
 319 02:00:50  <crimsun> correct
 320 02:00:55  <carthik> ok
 321 02:01:21  <crimsun> So, two remaining steps: 1) Generate the new merged source package, and 2) test-build it.
 322 02:02:14  <crimsun> (1) is done by invoking the merge-buildpackage script in /tmp/merges/
 323 02:02:24  <carthik> let's go for it then?
 324 02:02:33  <crimsun> I normally do: ../merge-buildpackage -rfakeroot -kC88ABDA3
 325 02:02:51  <crimsun> where -kC88ABDA3 is my gpg key id
 326 02:02:56  <carthik> the last bit being your key
 327 02:02:59  <crimsun> right
 328 02:03:00  <carthik> ok :)
 329 02:04:12  <crimsun> if you have pbuilder configured, you'll then want to test-build the merged dsc.
 330 02:04:41  <carthik> make: dh_testdir: Command not found
 331 02:04:50  <carthik> is that some package I missed out on installing?
 332 02:05:01  <crimsun> yeah, you'll need debhelper installed.
 333 02:05:08  <crimsun> dh_* are debhelper scripts.
 334 02:05:41  <carthik> okay this time it went through
 335 02:05:51  <carthik> I don't have pbuilder configured
 336 02:06:14  <crimsun> That will be covered in another session.
 337 02:06:19  <carthik> :) okay
 338 02:06:34  <crimsun> It's more important to know that you have to at least test-build your newly merged source package. :)
 339 02:06:37  <carthik> so you test-build it and then you're done?
 340 02:07:58  <crimsun> yes, then you can ping an MOTU to upload it.
 341 02:08:07  <crimsun> here's the log from pbuilder: http://pastebin.ca/104636
 342 02:08:32  <carthik> so what do I have have to upload and where?
 343 02:08:38  <carthik> and will that get reviewed then?
 344 02:09:51  <crimsun> you have a couple options: You can either upload to REVU (http://revu.tauware.de/), or you can file a bug using http://launchpad.net/ against the source package
 345 02:10:28  <carthik> okay, and upload what?
 346 02:10:33  <carthik> the .dsc ?
 347 02:10:41  <carthik> the new one that is
 348 02:10:44  <crimsun> It's immensely helpful to have your own Web space so that you can upload the .dsc and .diff.gz.
 349 02:11:11  <carthik> ah, okay, the .diff.gz too
 350 02:11:16  <crimsun> otherwise you'll upload the generated _source.changes, .dsc, .diff.gz, and .orig.tar.gz to REVU.
 351 02:11:22  <crimsun> (REVU is a topic for another session.)
 352 02:11:52  <crimsun> Now to tie this into current Edgy transitions, then to outline an algorithm.
 353 02:12:00  <carthik> one sec
 354 02:12:21  <carthik> what was the _source.changes about? is that an output of the build process?
 355 02:13:04  <carthik> I mean, where will these files all be, using /tmp/merges/ as the 'basedir'
 356 02:13:09  <crimsun> no, that's a listing of the changes and the source package that would be uploaded to the Ubuntu processing infrastructure.
 357 02:13:14  <carthik> in that same directory?
 358 02:13:19  <crimsun> they'll be in /tmp/merges/
 359 02:13:45  <carthik> but isn't it what we downloaded first - I mean - some of those files very never changed right?
 360 02:14:00  <carthik> s/very/were/
 361 02:14:01  <crimsun> specifically /tmp/merges/m2crypto_0.16-1ubuntu1.diff.gz, /tmp/merges/m2crypto_0.16-1ubuntu1.dsc, /tmp/merges/m2crypto_0.16-1ubuntu1_source.changes, and /tmp/merges/m2crypto_0.16.orig.tar.gz
 362 02:14:29  <crimsun> carthik: right, some of those files are only downloaded for references
 363 02:14:34  <carthik> okay cool
 364 02:15:18  <carthik> I suppose it will be okay to upload all these files to one's personal web-space -- just in case.
 365 02:15:22  <crimsun> carthik: ultimately the only ones that matter are the ones you regenerate, which constitutes the merged source package (.diff.gz + .dsc + _source.changes [+ possibly .orig.tar.gz])
 366 02:15:31  <crimsun> carthik: yes, that would be fine.
 367 02:16:17  <carthik> cool
 368 02:16:33  <carthik> I suppose MoM does save a lot of time for MOTUs
 369 02:16:34  <carthik> :)
 370 02:16:37  <crimsun> yes.
 371 02:16:55  <crimsun> Now you've just witnessed the reasoning behind the new Python Policy.
 372 02:17:16  <crimsun> i.e., changing stuff for Python versions 2.2, 2.3, 2.4, ... is tedious.
 373 02:17:52  <carthik> yes - this should be better :)
 374 02:18:33  <crimsun> That's why python-support and python-central are useful, and that's documented in the newer Python Policy at http://www.debian.org/doc/packaging-manuals/python-policy/
 375 02:19:16  <crimsun> Specifically, http://www.debian.org/doc/packaging-manuals/python-policy/ap-packaging_tools.html is relevant here.
 376 02:20:31  <crimsun> Which one you should choose is outside the scope of this session, which is already running a bit long. :)
 377 02:20:39  <carthik> yeah
 378 02:20:47  <crimsun> So, briefly, I'll outline an algorithm for tackling merges.
 379 02:22:01  <crimsun> Step one: Grab the Merge-O-Matic output. You can do this using grab-merge.sh. I'm omitting the details for configuring necessary packages like fakeroot, debhelper, devscripts, build-essential, cdbs, etc.
 380 02:22:55  <crimsun> Step two: Read the Merge-O-Matic output (either in the extracted source or via REPORT).
 381 02:23:56  <crimsun> Step three: Inspect the conflicts in each file listed in the M-O-M output, and follow this subroutine:
 382 02:24:46  <crimsun> Prefer the newer Debian changes when possible. This means we discard Ubuntu changes until we have to use one.
 383 02:26:16  <crimsun> If newer Debian changes were made that don't affect the old Ubuntu source package, then extract the newer pristine Debian source package and check if those Debian changes allow the Ubuntu merges to be dropped from the merged source.
 384 02:28:03  <crimsun> If those Debian changes allow the Ubuntu merges to be dropped, then make the modifications in the merged source package.
 385 02:28:23  <carthik> oh so check the /debian/patches in the new pristing Debian source to see if any of them duplicate the patches in the MOM Ubuntu source
 386 02:28:40  <carthik> duplicate or supercede... cool... got it now.
 387 02:28:55  <crimsun> (Correct for the previous example, but it won't always be debian/patches/ ; it could be plain source, which could be tedious.)
 388 02:29:04  <carthik> okay
 389 02:29:40  <carthik> (please carry on, and I'll hold my exclamations for when you say you're done)
 390 02:29:41  <crimsun> Continue this step for all files listed in the M-O-M output.
 391 02:31:11  <crimsun> Step four: Sanity-check the package. For instance, you don't want hardcoded paths for older software (that you just dropped) sitting in the source.
 392 02:32:21  <crimsun> Step five.: Change the debian/changelog to note your work. Note which Ubuntu deltas are still valid and why invalid ones were dropped. Change the M-O-M e-mail address and name to yours.
 393 02:32:48  <carthik> okay
 394 02:33:35  <crimsun> Step six: Generate a new merged source package using the merge-buildpackage script. Remember to pass any necessary options like " -rfakeroot -k<your_gpg_key_id> ".
 395 02:34:29  <crimsun> Step seven: Sanity-build the source package in pbuilder, check the contents of the created debs (using dpkg-deb -c foo.deb), and alert a MOTU to your merged source package.
 396 02:35:28  <crimsun> (End of sequence)
 397 02:36:14  <carthik> that was great, crimsun.
 398 02:37:12  <carthik> Things I've wanted to know about for some time now.
 399 02:37:26  <carthik> Sorry if I bugged you with too many stupid questions
 400 02:37:43  <crimsun> Merging is generally pretty straightforward. You don't have to have in-depth knowledge of packaging to help; it's mostly reading whether Ubuntu changes have been absorbed by Debian.
 401 02:37:57  <crimsun> No problem at all, questions are always welcome.
 402 02:38:03  <carthik> There was the one question about what to do to check if any of the new depends or build-depends packages force removal of some other package...
 403 02:38:16  <crimsun> Right.
 404 02:38:54  <carthik> exactly - so I thought if there was something relatively more mechanical than something that needs one to be a DD - that is something I can help out with
 405 02:40:01  <crimsun> xmakemol is a corner case: There aren't many packages that will require in-depth knowledge about whether libgl1-mesa-swx11 will force removal of libgl1-mesa-{dri,glx} .
 406 02:40:49  <carthik> okay
 407 02:41:10  <carthik> hopefully the reviewers will catch the slippery fish
 408 02:41:31  <crimsun> In this case, I felt it was a stronger justification to try and realign with Debian packaging, so libgl1-mesa-swx11-dev was preferred over xlibmesa-gl-dev.
 409 02:42:18  <crimsun> Arguably it's easier to add a Ubuntu delta than to comb through the M-O-M output to see if it can be dropped.
 410 02:43:13  <crimsun> Most of the time there's someone in #ubuntu-motu who will answer the more difficult dependency questions.
 411 02:43:39  <carthik> okay
 412 02:44:23  <carthik> Thanks crimsun, I am starving and it is 10:40 and there are 12 Jordanians waiting for me with beer and a hookah :)
 413 02:44:32  <crimsun> Does anyone (if anyone else's still awake) else have any questions?
 414 02:44:47  <crimsun> carthik: np, enjoy.
 415 02:45:05  <carthik> crimsun, I hope I can make all this worth your while - thanks again :)
 416 02:45:18  <crimsun> :)
 417 02:45:40  <crimsun> If no other questions, this concludes the merge section. An outline and log will be posted shortly.

Attached Files

To refer to attachments on a page, use attachment:filename, as shown below in the list of files. Do NOT use the URL of the [get] link, since this is subject to change and can break easily.
  • [get | view] (2006-08-03 06:51:18, 38.2 KB) [[attachment:crimsun-motu-school-trimmed.txt]]
 All files | Selected Files: delete move to page

You are not allowed to attach a file to this page.