Ubuntu Wiki

Ubuntu Open Week - Cruft. What is it and why it sucks - Michael Casadevall - Thu, Nov 6th, 2008

(03:01:05 PM) NCommander: Hello, and welcome to the Cruft 101 lesson here in #ubuntu-classroom
(03:01:34 PM) NCommander: I'm your host, NCommander, also known as Michael Casadevall and sonicmctails. I'm an MOTU, and the founder of the ubuntu-cruft-buster team on Launchpad
(03:02:10 PM) NCommander: As you are all still here, I expect your dying to learn the ins and outs of cruft in the Ubuntu archive
(03:02:30 PM) mode (-m ) by jcastro
(03:02:31 PM) ***NCommander waits for jcastro to -m the room
(03:02:33 PM) nizarus_ is now known as nizarus
(03:02:36 PM) NCommander: Ok, you can all talk freely :-)
(03:02:49 PM) NCommander: If you have any questions at a given moment, feel free to ask them
(03:03:20 PM) NCommander: Anyway, archive cruft is the Ubuntu equivalent to things like junk and other old pieces one might find in the closest
(03:03:22 PM) NCommander: *closet
(03:03:51 PM) NCommander: They don't directly hurt anyone, but it is undesirable, and removing cruft is a priority of all Ubuntu Developers, some more so than others
(03:04:22 PM) NCommander: Cruft, roughly defined, is a package or packages that are no longer used or needed in the Ubuntu archive
(03:04:37 PM) NCommander: In most cases, these are software libraries, but normal programs also are subject to cruft removal.
(03:05:20 PM) NCommander: Cruft accumulation is a normal part of software development, as software grows and matures, cruft will be generated
(03:05:29 PM) NCommander: But how it is generated is something we'll come back to in a minute
(03:06:00 PM) NCommander: To understand cruft, you need to understand what reverse-dependencies, or rdepends are
(03:06:27 PM) NCommander: For anyone who has ever installed a piece of software on Ubuntu, you know that one program is often dependent on other bits of software
(03:06:52 PM) NCommander: These dependent bits are a packages dependencies
(03:07:07 PM) NCommander: rdepends are packages that depend on one specific package
(03:07:52 PM) NCommander: For instance, X11 is an rdepends of GNOME, or XFCE, or KDE (this is on a very simple level, but the concept applies with concept management)
(03:08:39 PM) NCommander: To remove cruft from an archive, that piece of cruft must have no reverse-dependencies that aren't also being removed
(03:09:06 PM) NCommander: rdepends can be checked with the apt-cache command
(03:09:53 PM) NCommander: When you run apt-cache rdepends on a package, it lists all the reverse dependencies of that package, or none if there are none
(03:10:30 PM) Guest2365 is now known as WastePotato
(03:10:41 PM) NCommander: A good example is libwxgtk2.4, an old wxGTK library which is very close to be ready for removal from the archive is libwxgtk2.4-1
(03:10:46 PM) NCommander: On my system, I get the following output:
(03:10:52 PM) NCommander: mcasadevall@blacksteel:~/src/sponsoring$ apt-cache rdepends libwxgtk2.4-1
(03:10:52 PM) NCommander: libwxgtk2.4-1
(03:10:52 PM) NCommander: Reverse Depends:
(03:10:52 PM) NCommander:   python-wxgtk2.4
(03:10:52 PM) NCommander:   libwxgtk2.4-dev
(03:10:53 PM) NCommander:   libwxgtk2.4-1-contrib
(03:10:55 PM) NCommander:   ctsim
(03:11:38 PM) NCommander: As the wxgtk2.4 packages are slated for removal, that leaves ctsim the only package that would have to be migrated to a new wxGTK to remove 2.4 as cruft
(03:12:02 PM) NCommander: Now, by this point, you must be wondering how cruft ends up in the archive
(03:12:44 PM) NCommander: There are usually two main ways, when a package is updated, and a library's soname is bumped, OR when a libraries API changes to the point that we need a seperate package for that library
(03:13:15 PM) NCommander: sonames define the ABI, or Application Binary Interface of a library
(03:13:40 PM) NCommander: In laymans terms, the ABI is how other applications access functions and code of a library
(03:14:12 PM) NCommander: As long as an ABI is unchanged, any programs that link against that library will continue to work even when that library is updated
(03:14:37 PM) NCommander: I realize I'm going kinda fast, does anyone have any questions?
(03:14:44 PM) NCommander: (before I continue)
(03:15:15 PM) NCommander: Ok, I'm going to assume no questions and continue
(03:15:54 PM) NCommander: ABIs change whenever a upstream maintainer does something to change the prototypes of a library
(03:16:29 PM) NCommander: Prototypes are entry points into a library, they say how a program can access it, and how many bits of data it will accept via pointers.
(03:17:16 PM) NCommander: When these prototypes change, programs calling the old prototype fails
(03:17:26 PM) NCommander: (this sometimes shows up as Can not resolve symbol error)
(03:18:14 PM) NCommander: Prototypes change because a maintainer adds or removes a feature, or changes the size of the data a function can take (i.e., replacing a int with a long)
(03:18:59 PM) NCommander: When this happens, the library maintainer is supposed to bump the soname to show that an ABI break has happened, and any packages that rdepend on it must be transitioned
(03:19:39 PM) NCommander: Now if your exceptionally lucky, the upstream maintainer has not changed the Application Programming Interface, or API
(03:19:59 PM) NCommander: APIs are like blueprints, they tell a program, at compile time how to call a libraries methods and functions
(03:20:41 PM) NCommander: A more realistic scenario is that whatever application rdepended on your library, it will likely need tweaking (or in some cases, full out porting)
(03:21:13 PM) NCommander: Actually doing porting is outside the scope of this discussion, and in most cases unneccessarily fortunately
(03:21:21 PM) NCommander: (exceptions do exist, for instance gtk1.2 -> 2)
(03:22:12 PM) NCommander: Whenever an ABI break happens, we need to do a transition to move all packages from the old version of the library to the new
(03:22:35 PM) NCommander: These happen in two forms, NBS (Not Built From Source), or multiple libraries packaged seperately
(03:22:45 PM) NCommander: NBS is the most common, so we will look at that first
(03:23:07 PM) NCommander: Now for those of you who attended Packaging 101, you may be wondering how a package in Ubuntu can't be built from source
(03:23:46 PM) NCommander: The answer is your right, no package can exist without its assiocated source package. NBS more properly defined are Packages that are no longer built from current source packages
(03:24:10 PM) NCommander: For instance, lets say I have liba0 in the archive
(03:24:28 PM) NCommander: (all library packages have a libname0, and a libname-dev package, or something similar)
(03:24:43 PM) NCommander: If the upstream maintainer releases a new version with a soname break
(03:24:56 PM) NCommander: I'll now have a liba1 and a liba-dev, instead of a liba0, and a liba-dev
(03:25:22 PM) NCommander: Now since liba-dev depends on the new liba1, installing it pulls in the new package
(03:25:37 PM) NCommander: The old library, since it is no longer built from the liba source package becomes a NBS library
(03:26:34 PM) NCommander: In some cases, doing the transition is as trival as a no-changes upload to cause Launchpad to rebuild your package against the new API
(03:26:38 PM) NCommander: *ABI
(03:27:12 PM) NCommander: In other cases, you'll likely have to tweak the package to get it build, or (more likely) package a newer upstream or sync a newer upstream from Debian
(03:27:46 PM) NCommander: Once all rdepends on a NBS library are removed
(03:28:04 PM) NCommander: The library magicially go poof via an NBS cleaner script run by your handy dandy archive administrators
(03:28:11 PM) NCommander: (or in other words, they remove themselves)
(03:28:58 PM) NCommander: In some cases however, NBS is not possible or apporiate
(03:29:31 PM) NCommander: This is normally the case for packages with a tremendous number of rdepends, or ones who have API breaks to the point that each rdepend requires manual porting to get working
(03:29:47 PM) NCommander: (gtk1.2, and wxgtk2.4/2.6 are good examples of these in the archive)
(03:30:25 PM) NCommander: In that case, what will happen, you will end up with multiple binary and -dev packages
(03:30:43 PM) NCommander: To do the transition, each rdepend must be downloading, its build-dependencies changed, build testing, installed, etc.
(03:30:54 PM) NCommander: In some cases, this will be as change it and build it, with no other necessary changes
(03:31:31 PM) NCommander: In others, it may require working with upstream authors to port their code (or if the package is dead upstream, doing the work yourself for a cruft free future)
(03:32:26 PM) NCommander: Any questions so far?
(03:33:07 PM) NCommander: I'll assume no, and continue onward
(03:33:38 PM) NCommander: Once you have finished removing all the rdepends of a package, you need to actually remove it from the archive
(03:34:00 PM) NCommander: Unlike NBS packages, oldlibs do not automagicially remove themselves when their dependencies are removed
(03:34:33 PM) NCommander: Manual intervention from the archive administrators are required
(03:34:37 PM) NCommander: This is known as a Removal Request
(03:35:40 PM) NCommander: Removal Requests state if a source or binaries are being remove, what distributions to remove them from (usually only the current development), and why, as well as confirmed there are no rdepends left of that package
(03:36:02 PM) NCommander: As with all packages related tasks
(03:36:24 PM) NCommander: Removal requests must be done or at least ACKed by an MOTU for universe/multiverse, or a core developer for main/restricted
(03:36:40 PM) NCommander: (it should be noted that a direct removal from main is extremely rare, usually the package will be demoted to universe, then removed)
(03:37:32 PM) NCommander: Removals are a two step package
(03:37:34 PM) NCommander: er
(03:37:36 PM) NCommander: process :-)
(03:38:15 PM) NCommander: Beside the actual removal of the package from Ubuntu, the blacklist file may also be updated
(03:38:54 PM) NCommander: As anyone running Jaunty knows, packages are pulled from Debian for roughly the first three-four months of a development build cycle (this is known as autosync)
(03:39:04 PM) NCommander: The autosync blacklist does extactly what it says on the tin
(03:39:23 PM) NCommander: If a package is blacklisted, unless an MOTU or coredev do an explicate sync request that package will not be pulled over from Debian
(03:40:05 PM) NCommander: Speaking of Debian, I should briefly bring up how Debian does transitions
(03:40:20 PM) NCommander: Just like Ubuntu, Debian goes through transitions (all distributions do)
(03:40:50 PM) NCommander: Unlike Ubuntu, rebuilding binary packages on Debian do not require a no-changes source upload
(03:41:02 PM) NCommander: Debian does whats called a binNMU, or binary Non-Maintainer Upload
(03:41:14 PM) NCommander: These packages are versioned with a +bX in the deb
(03:41:53 PM) NCommander: For most intents and purposes, you can pretend these do not exist, but if your looking for hints on how to transition a package, seeing if Debian has done any recent binNMUs on said package or its rdepends is a good place to start
(03:41:59 PM) NCommander: (the changelog is also another good place to start)
(03:42:47 PM) NCommander: I've covered a lot of information thus far in the last 43 minutes
(03:43:03 PM) NCommander: I'd like to open the floor to any questions (I've covered all the basics, but I bet a bunch of you are scratching your heads)
(03:44:08 PM) sudobash: 3+.
(03:44:13 PM) NCommander: sudobash, shoot
(03:44:54 PM) sudobash: sorry accident...
(03:45:05 PM) ***NCommander facepalms
(03:45:54 PM) NCommander: Seriously, no questions? I finished my basic overview of cruft, and I was willing to work on doing some hands on cruft removal if anyone was interested
(03:46:33 PM) NCommander: Anyway, if you have any questions about cruft, feel free to ping me on IRC< or any of the cruft buster team members
(03:48:18 PM) artir: QUESTION: When removing cruft packages from user's pcs, what is used: apt-get remove or apt-get remove --purge ?
(03:48:33 PM) NCommander: artir, excellent question
(03:49:00 PM) NCommander: If your removing a cruft package directly, apt-get remove *cruftlib* works like you expect it to
(03:49:38 PM) NCommander: If a package has no dependencies is installed, and it was installed as part of another package, the next time apt-get remove --purge is run, it is removed
(03:50:39 PM) NCommander: artir, does that answer your question?
(03:50:56 PM) artir: yep, thanks :)
(03:51:25 PM) NCommander: MadsRH, you had a question?
(03:52:03 PM) MadsRH: QUESTION: Not a very technical question, but I was just looking at "Remove Cruft" in the menu and why did you decide to use that icon? I really don't see the link to removing cruft. Is it just a temporary icon or is it there to stay?
(03:52:30 PM) NCommander: MadsRH, where specifically is the Remove Cruft?
(03:52:32 PM) NCommander: In Synaptic?
(03:52:50 PM) jcastro: I think he's mixing up what your topic is with the cruft remover app that shipped with intrepid
(03:52:57 PM) NCommander: Oh!
(03:53:07 PM) MadsRH: system -> admin. -> remove cruft
(03:53:11 PM) NCommander: gtk-cruft-removal (now gtk-system-cleaner) is something completely different ;-)
(03:54:13 PM) MadsRH: Sorry :-[ Just jumped in!
(03:54:30 PM) NCommander: MadsRH, thats fine
(03:57:18 PM) NCommander: I guess that does it on cruft
(03:58:09 PM) ***knome applauds for NCommander 
(03:58:12 PM) NCommander: I did have some work showing actual examples of cruft hands of, if people are interested in that
(03:58:42 PM) RainCT: nice explanation, NCommander :)
(03:58:57 PM) charlie-tca: Thank you
(03:59:16 PM) artir: so the biggest problem to deal with cruft is to know what is, actually, cruft
(03:59:49 PM) NCommander: artir, with some experience, it becomes more clear
(04:00:00 PM) NCommander: also, there is an oldlibs category for packages that are cruft
(04:00:06 PM) NCommander: but still have rdepends thus unremovable