LibraryPackaging
This session was presented by Stefan Potyra and Steve Langasek on 18th January 2008.
20:06 sistpoty so a big welcome to everyone for joining the motu school session about library packaging 20:06 sistpoty the session will be split in two parts: 20:07 sistpoty in the first part, we'll dive deeply into the background of shared objects, and how to find out information about these 20:07 sistpoty and in the second part, we'll go through creating a library package by example 20:07 sistpoty so let's start with part 1 20:07 sistpoty First off, shared objects are useful for two different purposes 20:08 sistpoty 1) as a plugin mechanism in conjunction with dlopen 20:08 sistpoty 2) for dynamic libraries, to share code between different projects 20:08 sistpoty as this is about library packaging, we'll be only looking at 2) in this session 20:09 sistpoty First off, (ELF) shared objects are a collection of symbols in different sections together with some meta-information 20:09 sistpoty a symbol denotes a named entity in C (e.g. a function or a variable) 20:09 sistpoty the section will give us some detail about the symbol 20:10 sistpoty btw.: if you have questions, just ask ;) 20:10 sistpoty let's take a look at an example: http://www.potyra.de/library_packaging/example.c 20:10 sistpoty so please everyone download the file 20:11 sistpoty everyone got it? 20:11 warp10 yep! 20:11 mok0 + 20:11 sistpoty great, so let's compile the file: 20:11 sistpoty gcc -c example.c -o example.o 20:12 sistpoty so far, the .o is just a normal object, not a shared object yet... however we can do some stuff with it already 20:12 sistpoty now run 20:12 sistpoty nm example.o 20:12 sistpoty the output are the different symbols defined in example.c 20:14 mok0 what are the letters U, T, t... 20:14 sistpoty for example in line 3, the variable "static_global" is results as symbol "0000000000000000 d static_global" 20:14 sistpoty mok0: will talk about that in a second ;) 20:14 sistpoty when you compare the output of nm, together with the variables of example.c, does anyone see anything odd? 20:15 sistpoty I'll give you a hint: look at the beginning of the function local_function 20:16 sistpoty now, anyone? 20:16 warp10 why local_var isn't listed? 20:16 sistpoty warp10: excellent 20:16 mok0 it's allocated on the call stack 20:17 geser why does static_local_var has a number in the nm output (here: static_local_var.2268)? 20:17 sistpoty excactly... some symbols in C (like local_var) don't need to be stored anywhere, because these are located in the stack 20:18 sistpoty ok, and some variables (like static_local_var) won't result in the same symbols as they are named. This is because the variable is not visible anywhere but in the function, and you can name a variable like this anywhere else 20:18 slangasek geser: that's a very good question whose answer I don't remember :) 20:18 sistpoty and you cannot have two symbols with the same name of course 20:18 mok0 geser: could that number be the offset? 20:19 sistpoty any more questions or shall we go to the funny letters? 20:19 mok0 ^^ 20:19 slangasek mok0: it's not the offset 20:19 warp10 What's the order with which symbols are listed by nm? 20:19 sistpoty warp10: good question: slangasek? 20:20 slangasek plain alphabetical 20:20 siretart slangasek: in order to avoid name clashes between different static local variables with the same name in different functions? 20:20 sistpoty mok0: I guess it's just a counted number gcc will assign, but I may err on this side 20:20 mok0 sistpoty: my number is != gesers 20:20 geser mok0: it could be the line number after including all headers. after changeing the line with the previous one the number is now 2267 20:21 slangasek siretart: probably, yes 20:21 sistpoty mok0: he, ah 20:21 mok0 static_local_var.1781 20:21 slangasek siretart, geser: suffice to say, it goes away once you've linked the object into a shared library, so it ends up not being very relevant 20:22 sistpoty ok, then now let's take a look at the letters. 20:22 sistpoty First off, some are lower case and some are upper case. 20:22 slangasek geser: it isn't the line number; if I run the source through gcc -E, I get line 757 20:22 geser ok 20:22 sistpoty The lower case letters denominate local symbols. Noone except this very object file can see these kind of symbols 20:23 mok0 lower case == local, Upper case == global 20:23 sistpoty whereas the upper case letters mean that the symbol is global, and other object files are global... exactly. 20:23 sistpoty the letter denote the type of the symbol: 20:23 sistpoty t -> text section (may be mapped readonly), executable 20:23 sistpoty d -> initialized data (statics, initialized globals) 20:23 sistpoty (as in variables) 20:24 sistpoty c -> uninitialized data (uninitialized global variables) 20:24 sistpoty r -> read only data (const variables), not necessarily read only though. 20:24 slangasek (btw, if you later don't remember what these letters map to, they're listed in man nm) 20:24 sistpoty and then there's an interesting one: u 20:24 sistpoty this means that the symbol is not defined in the object, but somewhere else 20:25 sistpoty (undefined) 20:25 slangasek to clarify, that means the object *refers* to the symbol but does not define it 20:25 sistpoty so "stderr" is listed there, as it comes somewhere from the c library, as well as fprintf 20:26 sistpoty ok, now let's make a shared object from this file: 20:26 sistpoty gcc -Wl,-z,defs -Wl,-soname,libexample.so.1 -fPIC -shared example.c -o libexample.so 20:27 sistpoty everyone got it? 20:27 warp10 y 20:27 mok0 + 20:27 sistpoty ok, now let's look again at it with nm 20:28 warp10 wow... 20:29 sistpoty (you can ignore anything with an underscore, that's internal stuff) 20:29 bddebian nm is like objdump? 20:29 slangasek yes 20:29 bddebian ok 20:29 slangasek I might suggest at this point also running 'nm -D libexample.so', for comparison 20:29 sistpoty erm... sorry, we need to do another thing I forgot... let's first strip off the debugging symbols 20:30 sistpoty strip libexample.so 20:30 slangasek ... or that :) 20:30 geser nm: libexample.so: no symbols 20:30 mok0 hehe 20:31 mok0 that's easy :-) 20:31 sistpoty ok, to look at the dynamic symbols, you'll now want nm -D as slangasek suggested 20:32 slangasek right - since we strip binaries by default, 'nm foo.so' in the wild is going to give you empty output 20:32 geser the local symbols aren't listed anymore? 20:32 sistpoty what you'll see is that there are now (apart from some internals) only global symbols left. i.e. everything not defined as "static" 20:32 sistpoty exactly 20:33 sistpoty (side note: the static modifier in c means that the resulting symbol will be a local symbol) 20:34 sistpoty any questions so far? then let's try some other tools on the shared object 20:34 mok0 sistpoty: well, global to the functions in that file 20:34 sistpoty mok0: yes, a local global *g* 20:34 bddebian hrh 20:34 slangasek geser: since they're local, they're by definition not accessible to other things linking to this code; so aside from debugging symbols, which we've used 'strip' to wipe out, there's no reason to take up space by keeping around names for these variables 20:35 sistpoty ok, now let's take a look at the meta-information: 20:35 sistpoty objdump -x libexample.so 20:36 sistpoty this will give lots of info. The most interesting stuff can be found with objdump -p libexample.so as well 20:37 bddebian Who uses anything more than |grep SONAME? ;-) * slangasek raises his hand 20:37 sistpoty one thing that's in there is the SONAME, we'll take a look at it later 20:37 sistpoty the other thing in there is the NEEDED entry (or entries) 20:38 sistpoty the SONAME denotes the ABI-Verison of the shared object... 20:38 sistpoty everything with the same SONAME can be considered to be compatible 20:39 geser if upstream doesn't make a mistake 20:39 bddebian geser: Never happens ;) 20:39 slangasek everything with the same SONAME *will be* considered compatible by ld.so, so it darn well better *be* compatible :) 20:39 sistpoty geser: we'll just see what happens then ;) 20:39 sistpoty the NEEDED entries denote what libraries this shared object needs. These refer to the SONAME of other libraries 20:40 sistpoty I guess everyone knows ldd, let's try that one 20:40 sistpoty ldd libexample.so 20:41 sistpoty ldd will resolve any shared libraries a binary or a shared object needs 20:42 geser is linux-vdso.so.1 provided by the kernel? 20:42 sistpoty so, what's the difference between the NEEDED entries and the output of ldd apart from the pathname? 20:42 slangasek geser: yes, linux-vdso is a bit of Magic 20:43 warp10 ldd output shows three entries, not just one 20:43 geser ldd resolves them recursively * bddebian thinks geser is too smart for this class :) 20:43 sistpoty exactly what geser said :) 20:43 geser bddebian: :P 20:43 sistpoty so you should keep these in mind when dealing with shared objects, as you might want to look at the *direct* dependencies as well 20:44 slangasek yes; for a real world-example, have a look at ldd /usr/lib/libgnome-2.so.0, vs. objdump -p /usr/lib/libgnome-2.so.0 | grep NEEDED 20:45 slangasek so we've looked at the output of nm -D on this object, but nm -D has one weakness as a tool which is not at all obvious if you don't already know what's missing 20:46 slangasek have a look at the output of nm -D libexample.so again 20:46 slangasek now, compare it to the output of readelf -s libexample.so 20:46 slangasek who can spot the difference in the names? 20:47 bddebian The stuff from @@glibc? 20:47 geser you mean the versioned symbols? 20:47 slangasek yep - nm -D doesn't show symbol versions 20:47 mok0 cool 20:48 slangasek so if you want to get at those, you'll want to use objdump -T or readelf -s 20:48 geser what does the "(2)" in "6: 0000000000000000 144 FUNC GLOBAL DEFAULT UND fprintf@GLIBC_2.2.5 (2)" mean? 20:48 slangasek symbol versions are sometimes important, so you'll want to keep these tools in mind -- sometimes, an app/lib will not know the symbol version it's supposed to use, and that indicates a bug! 20:49 slangasek geser: I'm not sure, sorry 20:50 slangasek I'd probably have to look at the readelf source to figure that out, the fields aren't all well-explained in the documentation 20:50 mok0 we can write a dependency resolver from these tools 20:50 slangasek mok0: indeed, dpkg-shlibdeps uses these tools 20:51 sistpoty ok, any more questions so far? 20:51 sistpoty When writing a program, that uses a library, we want to make sure that the API of that library is stable. 20:52 sistpoty If we want to distribute this program, we also want a stable ABI. 20:52 sistpoty stable API means that my program compiles against the library and a newer version of it. 20:52 sistpoty stable ABI means that the symbols stay the same 20:53 mok0 what if there are new symbols? 20:53 sistpoty mok0: of course... I just wanted to correct myself 20:53 sistpoty stable ABI means that there won't get any symbols removed, and their meaning is still the same 20:55 mok0 If you write code that corrects a bug, but the interface remains the same? 20:55 sistpoty mok0: then the ABI and API is stable 20:56 sistpoty (most likely) 20:56 mok0 ... but I would still want that programs exclusively use the fixed version 20:56 sistpoty but let's first have some fun... as some newby upstream started to produce a library... 20:56 geser mok0: if I'm correct you can change the implementation as long as function name, arguments and return type stay the same 20:57 sistpoty mok0: you can do this (as upstream) only, if the library have some version string or date encoded 20:57 mok0 ok 20:57 sistpoty mok0: or by advetising it on your homepage *g* 20:57 mok0 you can bump the minor version number 20:58 sistpoty for the library: yes (I'll come to this part yet) 20:58 chiptoxic how do you tell if a library has a stable API or ABI is it just by watching the version numbers of that library? 20:58 sistpoty chiptoxic: I'd like to postpone this question, as I first want to show you an example 20:59 chiptoxic ok 20:59 sistpoty so please everyone get <http://www.potyra.de/library_packaging/libmyhello-1.0.1.tar.gz> 20:59 sistpoty if you've got it, extract it and compile it with 20:59 sistpoty make 21:00 sistpoty then install the result (as root) with 21:00 sistpoty make install 21:00 sistpoty (no worries, it will only copy files to the /usr/local/ namespace, and make uninstall removes it again) 21:01 sistpoty everyone got so far? (then me types the commands as well) 21:02 mok0 make PREFIX=. install 21:02 mok0 does it locally 21:03 sistpoty then you might have trouble upgrading the library 21:04 sistpoty (or building an application against it, which we'll do as soon as everyone is finished) * warp10 is done 21:04 sistpoty ok, next step: of course we want s.th. that uses the library, so get <http://www.potyra.de/library_packaging/hello_prog-1.0.tar.gz> 21:05 sistpoty extract that somewhere else and also build the binary with "make" 21:06 sistpoty if you've done that, you should have a cool binary hello_prog, which will print hello. you can even tell it how often it should print hello with the first argument 21:06 sistpoty ./hello_prog 10 21:07 sistpoty everyone got so far yet? 21:07 mok0 error while loading shared libraries: libmyhello.so.1: cannot open shared object file: No such file or directory 21:07 frigaut same here 21:07 geser mok0: sudo ldconfig 21:08 mok0 geser: just doing what the teachers tell me .-) 21:07 warp10 and here too :) 21:07 slangasek heh, did any of the people having this problem install to /usr/local? 21:08 slangasek or did you all install to a different prefix? 21:08 james_w slangasek: /usr/local for me 21:08 slangasek ah, ok 21:08 sistpoty hehe 21:08 frigaut works now after ldconfig 21:08 sistpoty of course I forgot to mention ldconfig 21:08 chiptoxic mine works and i just did what you said and ran ldconfig 21:08 sistpoty (still got it installed from testing *g*) 21:09 mok0 Yay, it says hello to me 10 times 21:09 kdub you have to have /usr/local in your /etc/ld.so.conf of course 21:09 geser sistpoty: "eventually run ldconfig now." 21:09 slangasek kdub: in /etc/ld.so.conf.d/libc.conf actually... :) 21:09 sistpoty geser: hehe, what a nice upstream *g* 21:10 geser yep 21:10 sistpoty now if it prints hello for everyone, let's remove the *library* again with 21:10 sistpoty make uninstall (run as root) 21:11 sistpoty now try again to run the hello_prog. 21:12 sistpoty any results yet? 21:12 geser ./hello_prog: error while loading shared libraries: libmyhello.so.1: cannot open shared object file: No such file or directory 21:12 mok0 boom 21:13 sistpoty great, so (I guess what you expected) it won't work w.o. the library 21:13 sistpoty now upstream has released a new version of the library... I guess you all want hot new stuff? 21:13 sistpoty then get it at http://www.potyra.de/library_packaging/libmyhello-1.0.2.tar.gz 21:13 sistpoty we'll be compiling it again: 21:13 sistpoty make 21:14 sistpoty and install it again (run as root) 21:14 sistpoty make install 21:14 mok0 If you install the library again, it works once more :-) 21:14 sistpoty hehe 21:14 warp10 I don't think so :) 21:15 geser ./hello_prog: symbol lookup error: ./hello_prog: undefined symbol: print_hello 21:15 sistpoty (oh, if you haven't removed installed the new library, you could also try ldd ./hello_prog) 21:16 sistpoty so what happened? 21:16 sistpoty now let's try to make use of the knowledge we gathered so far 21:16 mok0 the application attempts to use the new library 21:17 sistpoty mok0: exactly 21:17 sistpoty let's take a look at the symbols of both shared objects (old library vs. new library) 21:17 mok0 => ABI has changed 21:17 sistpoty excellent! 21:18 sistpoty in the old library, we had the function: T print_hello 21:18 sistpoty which upstream obviously renamed to just: T hello 21:19 sistpoty when you start the program, one of the first steps is that undefined symbols will get resolved * bddebian curses upstream 21:19 sistpoty hehe 21:20 slangasek yeah, I would think twice about packaging any libraries from this upstream ;) 21:20 sistpoty which is done recursively for each shared object that is used * sistpoty removes copyright notices *g* 21:20 mok0 This is why it is wrong to use the program version number for library versions 21:20 bddebian heh 21:20 mok0 which a lot of upstream programmers don't understand 21:21 sistpoty when the symbol lookup can't resolve one symbol.. well you just saw what happens. 21:21 sistpoty ok, what we've seen now is an incompatibility on symbol level. 21:22 sistpoty I hope with the tools you've learned so far, you can easily track those down 21:22 sistpoty -> any removed symbol in a shared library is a problem, *if* the SONAME stays the same 21:23 mok0 so let's change it 21:24 sistpoty mok0: feel free to change it, I guess you can figure this in the Makefile 21:24 mok0 I set major to 2 21:25 albert23 Wouldn't this be an API change, since print_hello is not defined anymore in hello.h? 21:25 albert23 I wouldn't expect this to compile anymore? 21:25 sistpoty albert23: exactly: this is an API change as well 21:26 sistpoty ok, but let's get back to the ABI... 21:27 sistpoty symbols however cannot represent the signature of functions (i.e. number of parameters, type of parameters) 21:27 sistpoty this is part of the ABI as well 21:27 sistpoty if we want to look if this changed, we'll need to dig a little bit deeper 21:27 mok0 Now the program works again, because it finds the first version of the library 21:27 mok0 (I changed MAJOR to 2) 21:28 sistpoty mok0: hehe, so you didn't unstall the old library? 21:28 chiptoxic im not a programmer but is there a way you could put an alias in the code say print_hello is really hello so you dont break the api? or would that just be messy 21:28 mok0 sistpoty: I still have 1.0.1 but I removed 1.0.2 21:28 mok0 The new one is 2.0.1 21:29 sistpoty chiptoxic: you could for example abuse the preprocessor: "define print_hello hello" 21:29 mok0 My experiment shows you can have several versions of the library present 21:29 sistpoty er... #define print_hello hello 21:29 chiptoxic ah cool, thanks 21:29 sistpoty ok, back to the topic... 21:30 sistpoty you can look at the debug information with gdb, so let's run 21:30 sistpoty gdb libmyhello.so 21:30 slangasek mok0: exactly, and this is a key reason why we always use the full soname for the names of our library packages 21:30 geser mok0: and now try the same on windows :) 21:30 sistpoty inside gdb, you can look at the signatures of functions with 21:30 sistpoty info functions 21:31 mok0 slangasek: you mean the major soname ? 21:31 slangasek mok0: the "soname" is the value returned by objdump -p $lib | grep SONAME 21:31 mok0 slangasek: ok 21:31 slangasek and set (hopefully) using the -Wl,-soname option at build time 21:32 sistpoty mok0: the Makefile just puts $(MAJOR) and a name in the soname, but the real soname is in SONAME 21:32 slangasek lintian knows the standard way to map sonames to package names, and warns you if your package is named differently. but of course, we have to take care that upstream is handling their ABI right in the first place! 21:32 sistpoty inside gdb, you can also display the types of variables with 21:32 sistpoty info variables === \sh is now known as \sh_away 21:33 sistpoty (which is uninteresting for libmyhello.so though, since it doesn't define a global variable by itself) 21:33 sistpoty everyone looked at the output of "info functions"? 21:34 mok0 just one: hello(void) 21:34 sistpoty you should see s.th. like "void hello(void);" or "void print_hello(void);" there (depending which version you chose) 21:34 sistpoty yes, so upstream is lazy as well *g* * bddebian files upstream bug 21:35 sistpoty now let's strip the shared object (which wasn't done during the build process): 21:35 sistpoty strip libmyhello.so 21:35 sistpoty (outside gdb again) 21:36 sistpoty and then call gdb libmyhello.so again and look at the functions again 21:36 mok0 :-( 21:36 sistpoty as you can see, these are all gone. 21:36 geser mok0: install the -dbgsym package of it :) 21:37 mok0 heh 21:37 sistpoty that's because these were part of the debug symbols, which have been stripped away 21:37 sistpoty ok, so if we want to find out about the ABI-part of the signatures, it's a little bit tougher, but still doable 21:39 sistpoty if a signature is changed (number of arguments and type of a function), it can lead to different results. Most probably it will segfault, since the library and the application no longer agree on what data they exchange) 21:39 sistpoty an exception to this rule is C++ 21:40 sistpoty in C++ the signature is encoded in the name of a symbol 21:40 sistpoty so if you change an argument, add an argument or remove one, you get a different symbol name 21:41 sistpoty (side note: you can use nm -C to demangle these names back to a human readable signature) 21:41 slangasek or c++filt 21:41 sistpoty any questions so far? 21:42 sistpoty chiptoxic: did that answer your question from some time ago? 21:42 slangasek other side note, checking for ABI changes of this kind by hand is hard and tedious; there is a tool in universe that tries to automate this checking, called icheck 21:42 chiptoxic yea, thanks 21:42 mok0 slangasek: icheck, what package? 21:43 sistpoty slangasek: ah, nice :) 21:43 sistpoty mok0: icheck 21:43 slangasek if you're maintaining a library and want to be careful because your upstream isn't, icheck will let you generate a "manifest" from the old and new -dev packages, and compare them to see if the signatures have changed * chiptoxic cant believe how much im learning :) 21:43 james_w slangasek: how does that compare to check-symbols from the ubuntu-dev-tools package? * mok0 install icheck 21:43 slangasek james_w: icheck examines headers, not symbols 21:44 slangasek james_w: so it's specifically for the case where you're trying to detect the kinds of ABI changes that /don't/ show up in the symbol table * bddebian can't keep up being at work :'-( 21:45 slangasek (doh, check-symbols uses nm -D? Baaad, what did we learn was the weakness of nm -D? :-) 21:45 sistpoty slangasek: since you mentioned ABI changes not showing up in the symbol table, want to say a few words to these? 21:46 slangasek probably, let me see where I put those words 21:47 slangasek right - does everyone here understand why changing the arguments or return type of a function is an ABI change? 21:47 mok0 yes 21:47 slangasek (if everyone knows, there's no sense in me spending time explaining it - but if you don't know, please speak up :) 21:48 warp10 y 21:48 mok0 Well renaming a global variable would show up 21:49 slangasek two people know who care, everyone else is asleep or doesn't want to talk to me ;) 21:49 slangasek sistpoty: back to you 21:49 slangasek :) 21:49 mok0 I just tried icheck on the 2 library versions, 21:50 mok0 but I couldn't get it to work 21:50 sistpoty ok 21:50 mok0 ... "Failed to parse libmyhello.so 21:50 sistpoty mok0: haven't used it myself yet 21:51 mok0 sistpoty: :-) 21:51 sistpoty what we should have learned so far is that if the ABI breaks, you'll need to bump the SONAME 21:51 slangasek mok0: you point it at the headers, not at the library 21:51 mok0 Ah! 21:51 sistpoty however there is yet another case, we didn't consider 21:51 slangasek that's the whole point - the library doesn't have the info you need about the function sigs :) 21:52 sistpoty upstream might put additional functionality in the library 21:52 sistpoty and keep the ABI downwards compatible 21:53 sistpoty in that case, the SONAME would (and should) stay the same, because no program ceases to work with the newer library 21:53 sistpoty however this has one caveat: people who use this library might make use of the new features 21:53 mok0 ... but new programs that use the new functionality? 21:54 sistpoty right, so these programs wouldn't work with the old library but only with the new one (even though the SONAME is the same) 21:54 sistpoty for this, there the SONAME is not sufficient alone. 21:55 sistpoty slangasek: told me that historically the minor version was incremented for this 21:55 sistpoty -: * mok0 also read that 21:55 sistpoty but that won't work if you copy binaries from one system to another (as you don't know about the libraries there) 21:56 sistpoty ok, this pretty much was part 1 -- shared objects 21:56 sistpoty any questions so far? 21:57 mok0 Well, what is the answer to the last problem? 21:57 sistpoty mok0: we'll see that in part 2 :) 21:57 mok0 Arrghh 21:57 warp10 :) 21:57 mok0 It stops at the most exiting part 21:58 mok0 typical of these mini-series :-)
20:06 sistpoty so who's around for part 2 of the library packaging session?
20:06 albert23 hello sistpoty
20:06 sistpoty hi albert23
20:07 sistpoty first off, there is an excellent guide about debian library packaging. We'll use this one as a reference for this session:
20:07 sistpoty http://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html
20:09 nixternal thanks sistpoty for the wakeup call :)
20:09 sistpoty hi nixternal :)
20:09 nixternal well howdy
20:09 sistpoty ok, in part 1, we hopefully learned all the basic stuff about shared objects.
20:10 ScottK You wouldn't have a link to the log for that would you?
20:10 nixternal I was just gonna ask, I have a lib I gotta package up :)
20:10 albert23 http://irclogs.ubuntu.com/2008/01/17/%23ubuntu-classroom.html
20:10 sistpoty ScottK: sorry.. no idea if this is publically logged somewhere
20:10 sistpoty thanks albert23
20:11 ScottK albert23: Thanks
20:11 sistpoty now in part 2, we'll be packaging a library by example, and will discuss a few of the caveats
20:12 sistpoty since I'm lazy, let's take libmyhello again from part 1
20:12 sistpoty http://www.potyra.de/library_packaging/libmyhello-1.0.1.tar.gz
20:13 sistpoty everyone got it?
20:13 warp10 y
20:13 hellboy195 y
20:13 albert23 yep
20:13 sistpoty ok, let's start with debianization... unpack it, rename the tarball so that we've got a valid .orig.tar.gz
20:14 sistpoty let's take a look, we'll always need a few files in there, let's start with debian/copyright
20:16 sistpoty put in the usual stuff there (this is not really important for the technical bits of library packaging)
20:17 sistpoty what else do we need... I guess we'll be using debhelper, so let's create a compat in debian/
20:17 sistpoty everyone got it so far?
20:17 hellboy195 y
20:17 warp10 y
20:18 albert23 y
20:18 sistpoty ok, now let's start with the interesting bits
20:18 sistpoty the control-file
20:19 sistpoty First off comes the usual stuff (I'll pastebin this)
20:19 sistpoty hm... pastebin is kinda slow today :(
20:20 sistpoty ok, I'll paste it right here... pastebin doesn't like me *g*
20:21 sistpoty Source: libmyhello
20:21 sistpoty Section: libdevel
20:21 sistpoty Priority: optional
20:21 sistpoty Maintainer: Your Name <youraddress@bla.org>
20:21 sistpoty Build-Depends: debhelper (>> 5.0.0)
20:21 sistpoty Standards-Version: 3.7.2.0
20:22 sistpoty for a library, we'll always end up with two packages
20:22 sistpoty two binary packages even
20:22 sistpoty one package contains the shared object(s)
20:23 sistpoty and another one, the -dev package will contain everything you need to develop applications with the shared object
20:24 sistpoty i.e. header-files, eventually a statically linked library, eventually pkconfig files (which specify what to pass to the compiler/linker to link against this library)
20:25 sistpoty so let's create an entry for the -dev package
20:25 sistpoty Package: libmyhello-dev
20:25 sistpoty Architecture: any
20:25 sistpoty Description: myhello library headers
20:25 sistpoty This package contains the development headers
20:25 sistpoty for libmyhello.
* ScottK notes sistpoty needs to update to the current standards version...
20:26 sistpoty ScottK: hehe, I'm not up to date
20:27 sistpoty the package containing the shared object will need to be named in a special way:
20:28 sistpoty it must be related to the soname.
20:28 sistpoty the library packaging guide I mentioned contains the rules how it should be named
20:29 sistpoty chapter 5, section 3
20:29 sistpoty remember the soname of libmyhello yet?
20:30 sistpoty anyone?
20:30 warp10 mmm... libmyhello.so.1, maybe?
20:30 sistpoty warp10: exactly
20:31 sistpoty if you wouldn't have remembered, you could of course always use objdump -p <shared object> to find out
20:33 sistpoty so in this case, the name of the library would be libmyhello1
20:33 sistpoty if you've got a library, that contains more than one shared object
20:33 sistpoty you'll need to create a binary package for each of these
20:33 sistpoty let's add an entry for the binary package
20:34 sistpoty Package: libmyhello1
20:34 sistpoty Architecture: any
20:34 sistpoty Depends: ${shlibs:Depends}
20:34 sistpoty Description: myhello library
20:34 sistpoty This package contains the hello library.
20:35 sistpoty ok. anyone got a clue, why the name of this binary must be related to the SONAME?
20:35 sistpoty this binary package...
20:36 sistpoty anoyne`
20:36 sistpoty +?
20:36 albert23 we want to be able to install multiple versions of the library, when ABI changes
20:36 sistpoty albert23: excellent
20:37 sistpoty so does anyone have an idea why this could be needed?
20:38 warp10 sistpoty: maybe different packages may need different versions of the library?
20:38 sistpoty warp10: this would be one argument, but not the main one.
20:38 sistpoty let's just consider that a very essential library (maybe libc) will get updated
20:39 sistpoty currently, I've got libc6 installed, and I guess almost all packages I've got installed will depend on libc6
20:39 sistpoty suddenly, a new libc7 appears
20:40 sistpoty if it couldn't be installed side-by-side to libc6, that would mean that I had to remove *all* installed programs
20:40 sistpoty or that all programs would immediately break if libc6 were removed by force
20:40 sistpoty that's pretty bad, hm?
20:41 sistpoty (I tried this once when I didn't have a clue... wasn't so funny back then *g*)
20:41 warp10 :)
20:41 sistpoty any questions so far?
20:42 warp10 I have one
20:42 warp10 Why the -dev package doesn't need any Depends: field?
20:42 sistpoty warp10: excellent question... it needs one :)
20:43 sistpoty so, as I wrote earlier, the -dev package is what you install if you want to build programs using this library
20:44 sistpoty so we need to make it depend at least on the library package
20:44 sistpoty let's add the depends line there
20:45 sistpoty the header files in question should always match what's in the shared object
20:45 sistpoty hence we'll need to restrict the version for this dependency
20:46 sistpoty Depends: libmyhello1 (= ${binary:Version})
20:47 sistpoty however often you'll need additional libraries to compile a program with a specific library
20:47 sistpoty so you'll need to add the corresponding -dev packages there as well.
20:48 sistpoty a good indication should give you the NEEDED entries of the shared object
20:49 sistpoty so you'll just need to find the corresponding -dev package that contains these shared objects
20:50 sistpoty so, what do we need here?
20:51 albert23 NEEDED libc.so.6
20:51 sistpoty great.
20:52 sistpoty so let's find out in which package this is (we'll see a better method later)
20:52 sistpoty just use dpkg -S libc.so.6
20:53 sistpoty hooray, it's libc6 :)... so we'll want libc6-dev
20:54 sistpoty (libc6-dev is often ommitted, since build-essential already depends on it. however it's not wrong to add it)
20:55 sistpoty as you might have noticed, in the case of libc6, the -dev package is not named like the source package
20:55 sistpoty but instead it is named like the binary package + -dev
20:56 sistpoty this is useful, if you want to have more than just one version of the library in the archive
20:56 sistpoty (e.g. because one application needs the old one, and won't build with the new version)
20:56 sistpoty then you'll need to be able to select which -dev package you want
20:57 ScottK There's a very recent thread on adding libc6 on debian-devel that came down pretty strongly in favor of adding it.
20:57 sistpoty if you do this, you should have the -dev package(s) of the different versoin Provide and Conflict to libmyhello-dev package
20:58 sistpoty ScottK: ah, haven't read that one yet
20:58 sistpoty (or maybe I have but forgot *g*)
20:59 sistpoty ok, does everyone have the control file so far?
20:59 albert23 y
20:59 warp10 y
21:00 sistpoty hehe, mine is still missing the depends line *g*
21:01 sistpoty ok
21:01 sistpoty what else do we need... a changelog file. just create one with dch
21:03 sistpoty dch --create
21:03 sistpoty (needed to look this up, thought it always was -c)
21:04 sistpoty everyone got it filled in?
21:04 warp10 y
21:05 albert23 y
21:05 sistpoty ok, then let's start with the rules file
21:06 sistpoty let's see what targets we need to adhere to the policy: clean, build, binary, binary-arch, binary-indep
21:06 sistpoty did I miss one?
21:07 warp10 install?
21:07 ScottK configure?
21:07 slangasek not required by policy :)
21:07 sistpoty yep
21:07 warp10 configure should be optional, right?
21:07 sistpoty hi slangasek btw.
* slangasek waves
21:08 warp10 install is optional too?
21:08 sistpoty warp10: libmyhello fortunately doesn't need to be configured, as it doesn't use autotools
21:08 sistpoty warp10: yes
21:09 sistpoty let's start with clean, should be easiest
21:09 sistpoty since we'll use debhelper, we'll first call dh_clean
21:09 sistpoty and then: make clean
21:09 sistpoty to cleanup libmyhello stuff
21:09 sistpoty or, if you prefer make distclean
21:10 sistpoty (which won't make a difference *here*)
21:12 sistpoty then let's get a shot at the build rule
21:12 james_w sistpoty: dh_testroot to start clean as well?
21:13 sistpoty james_w: right, makes sense
21:14 sistpoty ok, build should be easy as well.
21:14 sistpoty just a call to make
21:14 sistpoty any questions so far?
21:15 warp10 n
21:15 albert23 n
21:15 sistpoty ok, then let's go to the binary target
21:16 sistpoty since this is an arch:any package, we'll use the binary-arch target for the job, and have the binary target depend on it
21:17 sistpoty let's add some debhelper stuff
21:18 sistpoty dh_testdir
21:18 sistpoty dh_testroot
21:18 sistpoty dh_installchangelogs
21:18 sistpoty dh_installdocs
21:19 sistpoty dh_install
21:19 sistpoty dh_strip
21:19 sistpoty dh_compress
21:19 sistpoty dh_fixperms
21:20 sistpoty and now an interesting one: dh_makeshlibs
21:20 sistpoty (we'll have a look at it in a few minutes)
21:20 sistpoty dh_shlibdeps
21:20 sistpoty dh_installdeb
21:20 sistpoty dh_gencontrol
21:20 sistpoty dh_md5sums
21:20 sistpoty dh_builddeb
21:21 sistpoty since we'll need to build s.th. before we can install it, we'll add a dependency to build for binary-arch
21:22 sistpoty (a nicer way is to use stamps, but that's a different topic)
21:22 sistpoty now what does dh_makeshlibs do...
21:23 sistpoty as you may have noticed, whenever you put ${shlibs:Depends} in the control file, it will get resolved to the library packages during build
21:23 sistpoty this is done via the shlibs-mechanism
21:23 sistpoty if you've been here in part 1, I guess you could build one yourself
21:24 sistpoty if you look at /var/lib/dpkg/info, you'll find a number of shlibs-files
21:25 sistpoty let's take a look at one
21:26 sistpoty the format is: library-name soname-version dependencies
21:27 sistpoty let's look at /usr/lib/dpkg/info/libgnome2-0.shlibs
21:28 sistpoty compare this to the SONAME of /usr/lib/libgnome-2.so.0
21:28 sistpoty this explains the first two fields
21:29 sistpoty the third one is the package where this shared object can be found in
21:29 sistpoty the job of dh_makeshlibs is to generate such a file for us
21:29 warp10 cool...
21:31 sistpoty and at this point, we can try to answer mok0's question of part 1, what to do if upstream released a new abi compatible version, but programs use this one
21:31 sistpoty since the shlibs file can specify a version string (just look at the gnome shlibs file again)
21:31 sistpoty we can request to have a minimum version of the library
21:32 sistpoty this will get replaced for all packages that use ${shlibs:Depends} and build against our library
21:33 sistpoty so we only need to tell dh_makeshlibs, that a new feature landed in the library, so that it will set version field in the shlibs file right
21:33 sistpoty you can do this with dh_makeshlibs -V<version>
21:33 sistpoty now if anyone has installed an older version of the library, and wants to use the program that needs the new cool features
21:34 sistpoty he will have to upgrade the library via apt
21:34 sistpoty any questions so far?
21:34 albert23 How can we know which library version was the first providing a specific ABI version?
21:35 albert23 Isn't there a risk we set version to high in dh_makeshlibs -V<version>?
21:35 sistpoty albert23: pretty much through the same means that we can find out if the ABI is broken
21:35 albert23 ok
21:36 sistpoty albert23: there is no real risk in Ubuntu to set the version to high
21:36 sistpoty however setting the version not high enough can lead to problems
21:36 albert23 It would make backports more difficult?
21:37 sistpoty not too sure... I don't think so
21:37 sistpoty a backport gets compiled to whatever is in gutsy or feisty
21:37 sistpoty so it will get built against the feisty/gutsy version of the library, which has a different version requirement
21:37 ScottK So if you set it correctly and there isn't a sufficient ABI available, the backport build will, correctly, fail.
21:38 sistpoty well... it doesn't really relate to building, but rather to installin
21:38 sistpoty +g
21:39 sistpoty in order to build against a library, the version (and ABI) is irrelevant.
21:39 sistpoty what matters to build against it is the API
21:39 sistpoty the version rather enforces that users *need* to upgrade the library to a minimum ABI version
21:39 ScottK I guess I meant API then.
21:40 sistpoty so if the API changed, the field won't really be much of a problem, since this means a new ABI in almost all cases
21:40 sistpoty and hence a new SONAME, and a different library package
21:41 sistpoty this version is only relevant if the ABI/API stay the same, but new features got added
21:41 ScottK Ah. RIght.
21:41 sistpoty any further questions?
21:42 warp10 n
21:42 albert23 n
21:42 sistpoty ok, then let's try to fix the rules file and finish the package, shall we?
21:43 sistpoty since we use dh_install, we still need to install the files into debian/tmp/
21:43 sistpoty let's use PREFIX=$(pwd)/debian/tmp make install for this
21:44 sistpoty if you want, you can put this right at the beginning of binary-arch
21:44 sistpoty or you could add an install rule and have binary-arch depend on it... whatever you prefer
21:46 sistpoty now we'll need to sort which files go into which package
21:47 sistpoty so we want libmyhello1.install and libmyhello-dev.install
21:47 sistpoty the header files will go into the -dev package
21:49 sistpoty bah... back a few lines... (recalled the Makefile wrongly)... PREFIX should of course be $(pwd)/debian/tmp/usr
21:49 sistpoty I guess I should have added DESTDIR or s.th. to libmyhello's makefile
21:50 sistpoty in case we'd have a statically linked library (libmyhello.a), that would go into the -dev package as well.
21:51 sistpoty (only useful if you want to link s.th. statically against it)
21:52 sistpoty what else do we need in the -dev pacakge
21:52 sistpoty the .so symlink should go in there as well
21:53 sistpoty btw: the .so symlink is what the linker will look for if you give the -l parameter
21:53 sistpoty more specifically, if you'd call -lmyhello, it would prepand a lib and add an .so and look for this file (libmyhello.so)
21:54 sistpoty in case there are any other files for development purposes (pkconfig files for example), these should also go in the -dev package
21:54 sistpoty luckily we don't have these
21:56 sistpoty in the library package should be the shared object (of course) and the symlink that's named like the soname
21:56 sistpoty while the symlink itself would also be recreated by ldconfig
21:57 sistpoty including it in the library package has the advantage that it will get deleted if the library is removed
21:57 sistpoty because it's a file of the package
21:58 sistpoty everyone got the .install files?
21:58 albert23 y
21:58 warp10 y
21:59 sistpoty ok, let's give it a shot, and see if it works
21:59 sistpoty just use
21:59 sistpoty fakeroot make -f debian/rules binary
22:00 sistpoty to fix the errors
22:00 sistpoty (like my install files, that where wrong *g*)
22:00 albert23 mine too
22:00 warp10 argh... :-S
22:01 sistpoty did you fix the PREFIX=... make install?
22:01 albert23 y
22:01 warp10 sistpoty: just a wrong install for me too :)
22:01 sistpoty warp10: hehe
22:04 albert23 I think $(pwd) did not work
22:04 albert23 make tried to install in /debian
22:05 sistpoty it should install into debian/tmp/usr/
22:05 albert23 I got this: PREFIX=/debian/tmp/usr make install
22:06 sistpoty heh, I've got the same which doesn't work as well. but the answer is one line above
22:06 sistpoty PREFIX=debian/tmp/usr make install
22:06 sistpoty those nasty slashes *g*
22:06 warp10 grrrr
22:06 albert23 Yep, just removed the pwd part, that works
22:09 sistpoty hm... I guess I'll need to read the make manual again, I was so sure that $(PWD) exists (or $(pwd))
22:09 albert23 $(CURDIR) ?
22:09 slangasek $(PWD) exists only if it's set in the environment by the invoking shell
22:10 slangasek $(shell pwd) execs out to call the pwd command, so is rather inefficient
22:10 slangasek $(CURDIR) is the make builtin variable you should use
22:11 sistpoty thanks... I guess I confused this with $(pwd), which I often type in the shell *g*
22:11 slangasek :)
22:11 sistpoty so does everyone's build succeed now?
22:12 albert23 yep, package looks good
22:12 warp10 looks fine!
22:12 sistpoty damn, only mine doesn't build yet *g*
22:13 sistpoty ok, so you've got a source package and a binary
22:13 sistpoty let's run lintian at the binary packages. it will take a second look that we've got the files sorted out correctly
22:14 albert23 clean
22:15 sistpoty excellent!
22:15 sistpoty any questions so far?
22:16 albert23 no, that's pretty clear
22:17 warp10 it's fine. Just wondering if it would be so easy in the real-world packaging :)
22:18 albert23 I guess that strongly depends on upstream
22:18 sistpoty ok, so now what would you do if the ABI of a library changed, but the SONAME didn't?
* slangasek smiles
22:19 albert23 complain at upstream
22:19 sistpoty albert23: great, that should always be the first choice
22:19 sistpoty can you imagine, while it's a bad idea, if you bump the SONAME on your own?
22:20 sistpoty (as a packager)?
22:20 albert23 could be risky, as it might conflict with new upstream versions?
22:21 sistpoty that's one argument
22:22 sistpoty but you could append a debian or ubuntu somewhere and change the build scripts, so you could circumvent this
22:22 sistpoty any other arguments against this?
22:22 slangasek it leads to binary incompatibility with other distros
22:22 slangasek but that shouldn't outweigh the problems caused by binary incompatibility with previous releases
22:23 sistpoty also, it leads to binary incompatibility for binary only software
22:23 sistpoty (just imaging to change the SONAME of libc, and look at what breaks in multiverse)
22:24 slangasek well, a workaround for such binary-only software would be to instruct users to create their own local symlinks for the wrong soname
22:25 sistpoty heh
22:25 sistpoty ok, what other interesting points do we have...
22:25 slangasek like I said, the alternative is that you have to break all the /free/ software using the old version on upgrade :)
22:25 sistpoty yeah, it's either bad or worse *g*
22:26 sistpoty ok, what should you do when you upgrade a library?
22:26 slangasek by "upgrade", you mean "package a new version"?
22:26 sistpoty yes
22:27 sistpoty we already learned that we should look out for ABI compatibility, so let's look at the case where we *do* have a new SONAME
22:27 sistpoty and hence a new library package
22:28 sistpoty anyone any ideas?
22:28 slangasek grep -r $oldsover debian/ :)
22:28 albert23 I think we are going to have a problem with the -dev package
22:29 sistpoty albert23: what do you think could get problematic?
22:29 albert23 packages requiring the old ABI will get the new ABI when we would want to rebuild?
22:30 sistpoty exactly (but it's API in this case... compiling *P*rograms->aPi, binaries->aBi)
22:31 sistpoty so you should check which packages actually build-depend on the -dev package.
22:31 sistpoty if there is a large number, you could do local test-builds and see which fail
22:32 sistpoty in case you cannot live yet with the new version alone, you could for example make a different source package and hence have two version of the library in the archive
22:33 sistpoty one thing you might stumble over when dealing with libraries is a lintian warning about rpath
22:34 sistpoty when you link against a library, that has an rpath set, it will tell the loader to look for the shared object only in that very directory
22:34 sistpoty so you can specify -Wl,-rpath=/somewhere and link hello_prog against it
22:35 sistpoty then the loader will look for libmyhello only in /somewhere when hello_prog is started
22:35 sistpoty can anyone imagine, why using rpath for a library is a bad idea?
22:38 albert23 well, software would break if for some reason the library is moved to another directory....
22:38 sistpoty exactly
22:38 sistpoty and one use-case for moving a library is for example the ia32-libs package on amd64
22:39 sistpoty ok, any questions?
22:39 slangasek (and multiarch! whoo!)
22:40 slangasek albert23, sistpoty: I would note that the general case is that we don't want to continue supporting an old API for a library, even if it means some work has to be done to get the applications ported
22:40 slangasek so it's /usually/ better to have the new library version take over the source package name and the -dev package name, and only reintroduce an oldlibs package as a last resort
22:41 sistpoty thanks for clarifying
22:41 albert23 Well, one question: what would be the best place to look for bugs / qa issues to practice what we learned?
22:42 sistpoty hm... that's a good question indeed
22:42 sistpoty well, one thing is that you could review library packages on REVU
22:43 sistpoty or get your hands dirty and package a library
22:44 sistpoty ok, I guess we're through with the session... slangasek: anything important I forgot or you'd like to add?
22:45 slangasek sistpoty: I only had half an eye on the channel today; I guess you didn't talk about techniques maintainers can use to automatically detect ABI breakage when moving to a new upstream version?
22:45 sistpoty slangasek: nope
22:46 slangasek the dpkg in hardy includes the new dpkg-gensymbols code developed by Raphaël Hertzog in Debian; it basically creates a manifest, declaring which symbols were added in which version of the package
22:46 slangasek the /side/ benefit of this is that if you already know which symbols are supposed to be present, you know if any have gone missing
22:47 slangasek so http://qa.debian.org/cgi-bin/mole/seedsymbols is a site that includes automatically generated symbol manifests for all libs in the archive
22:47 slangasek (the Debian archive, that is; most will be applicable to Ubuntu too, obviously)
22:48 slangasek http://wiki.debian.org/UsingSymbolsFiles is a general description of this
22:48 slangasek I still need to use this on my own packages, so I can only give you theoretical insight into this stuff, not much practical :)
22:49 slangasek and everyone's eyes glaze over again ;)
22:50 albert23 Yeah, I guess I will first do some attempts to do the work manually.
22:50 albert23 I have bookmarked those links for later use...
22:51 slangasek one example of this in practice is the zlib1g package
22:51 slangasek which the maintainer, Mark Brown, blogged about as well (syndicated on planet.d.o)
22:52 sistpoty heh, I skipped this post, and wanted to read it later (which I then forgot *g*)
22:52 sistpoty ok, anything else?
22:53 sistpoty then thanks everyone for coming :)This log has been lightly edited for clarity.
MOTU/School/LibraryPackaging (last edited 2008-08-06 16:33:17 by localhost)