LibraryPackaging

Differences between revisions 1 and 2
Revision 1 as of 2008-01-19 09:52:22
Size: 49376
Editor: 77-99-12-164
Comment: Add the logs of the library packaging session.
Revision 2 as of 2008-08-06 16:33:17
Size: 49376
Editor: localhost
Comment: converted to 1.6 markup
No differences found!

This session was presented by Stefan Potyra and Steve Langasek on 18th January 2008.

20:06   sistpoty        so a big welcome to everyone for joining the motu school session about library packaging
20:06   sistpoty        the session will be split in two parts:
20:07   sistpoty        in the first part, we'll dive deeply into the background of shared objects, and how to find out information about these
20:07   sistpoty        and in the second part, we'll go through creating a library package by example
20:07   sistpoty        so let's start with part 1
20:07   sistpoty        First off, shared objects are useful for two different purposes
20:08   sistpoty        1) as a plugin mechanism in conjunction with dlopen
20:08   sistpoty        2) for dynamic libraries, to share code between different projects
20:08   sistpoty        as this is about library packaging, we'll be only looking at 2) in this session
20:09   sistpoty        First off, (ELF) shared objects are a collection of symbols in different sections together with some meta-information
20:09   sistpoty        a symbol denotes a named entity in C (e.g. a function or a variable)
20:09   sistpoty        the section will give us some detail about the symbol
20:10   sistpoty        btw.: if you have questions, just ask ;)
20:10   sistpoty        let's take a look at an example: http://www.potyra.de/library_packaging/example.c
20:10   sistpoty        so please everyone download the file
20:11   sistpoty        everyone got it?
20:11   warp10  yep!
20:11   mok0    +
20:11   sistpoty        great, so let's compile the file:
20:11   sistpoty        gcc -c example.c -o example.o
20:12   sistpoty        so far, the .o is just a normal object, not a shared object yet... however we can do some stuff with it already
20:12   sistpoty        now run
20:12   sistpoty        nm example.o
20:12   sistpoty        the output are the different symbols defined in example.c
20:14   mok0    what are the letters U, T, t...
20:14   sistpoty        for example in line 3, the variable "static_global" is results as symbol "0000000000000000 d static_global"
20:14   sistpoty        mok0: will talk about that in a second ;)
20:14   sistpoty        when you compare the output of nm, together with the variables of example.c, does anyone see anything odd?
20:15   sistpoty        I'll give you a hint: look at the beginning of the function local_function
20:16   sistpoty        now, anyone?
20:16   warp10  why local_var isn't listed?
20:16   sistpoty        warp10: excellent
20:16   mok0    it's allocated on the call stack
20:17   geser   why does static_local_var has a number in the nm output (here: static_local_var.2268)?
20:17   sistpoty        excactly... some symbols in C (like local_var) don't need to be stored anywhere, because these are located in the stack
20:18   sistpoty        ok, and some variables (like static_local_var) won't result in the same symbols as they are named. This is because the variable is not visible anywhere but in the function, and you can name a variable like this anywhere else
20:18   slangasek       geser: that's a very good question whose answer I don't remember :)
20:18   sistpoty        and you cannot have two symbols with the same name of course
20:18   mok0    geser: could that number be the offset?
20:19   sistpoty        any more questions or shall we go to the funny letters?
20:19   mok0    ^^
20:19   slangasek       mok0: it's not the offset
20:19   warp10  What's the order with which symbols are listed by nm?
20:19   sistpoty        warp10: good question: slangasek?
20:20   slangasek       plain alphabetical
20:20   siretart        slangasek: in order to avoid name clashes between different static local variables with the same name in different functions?
20:20   sistpoty        mok0: I guess it's just a counted number gcc will assign, but I may err on this side
20:20   mok0    sistpoty: my number is != gesers
20:20   geser   mok0: it could be the line number after including all headers. after changeing the line with the previous one the number is now 2267
20:21   slangasek       siretart: probably, yes
20:21   sistpoty        mok0: he, ah
20:21   mok0    static_local_var.1781
20:21   slangasek       siretart, geser: suffice to say, it goes away once you've linked the object into a shared library, so it ends up not being very relevant
20:22   sistpoty        ok, then now let's take a look at the letters.
20:22   sistpoty        First off, some are lower case and some are upper case.
20:22   slangasek       geser: it isn't the line number; if I run the source through gcc -E, I get line 757
20:22   geser   ok
20:22   sistpoty        The lower case letters denominate local symbols. Noone except this very object file can see these kind of symbols
20:23   mok0    lower case == local, Upper case == global
20:23   sistpoty        whereas the upper case letters mean that the symbol is global, and other object files are global... exactly.
20:23   sistpoty        the letter denote the type of the symbol:
20:23   sistpoty        t -> text section (may be mapped readonly), executable
20:23   sistpoty        d -> initialized data (statics, initialized globals)
20:23   sistpoty        (as in variables)
20:24   sistpoty        c -> uninitialized data (uninitialized global variables)
20:24   sistpoty        r -> read only data (const variables), not necessarily read only though.
20:24   slangasek       (btw, if you later don't remember what these letters map to, they're listed in man nm)
20:24   sistpoty        and then there's an interesting one: u
20:24   sistpoty        this means that the symbol is not defined in the object, but somewhere else
20:25   sistpoty        (undefined)
20:25   slangasek       to clarify, that means the object *refers* to the symbol but does not define it
20:25   sistpoty        so "stderr" is listed there, as it comes somewhere from the c library, as well as fprintf
20:26   sistpoty        ok, now let's make a shared object from this file:
20:26   sistpoty        gcc -Wl,-z,defs -Wl,-soname,libexample.so.1 -fPIC -shared example.c -o libexample.so
20:27   sistpoty        everyone got it?
20:27   warp10  y
20:27   mok0    +
20:27   sistpoty        ok, now let's look again at it with nm
20:28   warp10  wow...
20:29   sistpoty        (you can ignore anything with an underscore, that's internal stuff)
20:29   bddebian        nm is like objdump?
20:29   slangasek       yes
20:29   bddebian        ok
20:29   slangasek       I might suggest at this point also running 'nm -D libexample.so', for comparison
20:29   sistpoty        erm... sorry, we need to do another thing I forgot... let's first strip off the debugging symbols
20:30   sistpoty        strip libexample.so
20:30   slangasek       ... or that :)
20:30   geser   nm: libexample.so: no symbols
20:30   mok0    hehe
20:31   mok0    that's easy :-)
20:31   sistpoty        ok, to look at the dynamic symbols, you'll now want nm -D as slangasek suggested
20:32   slangasek       right - since we strip binaries by default, 'nm foo.so' in the wild is going to give you empty output
20:32   geser   the local symbols aren't listed anymore?
20:32   sistpoty        what you'll see is that there are now (apart from some internals) only global symbols left. i.e. everything not defined as "static"
20:32   sistpoty        exactly
20:33   sistpoty        (side note: the static modifier in c means that the resulting symbol will be a local symbol)
20:34   sistpoty        any questions so far? then let's try some other tools on the shared object
20:34   mok0    sistpoty: well, global to the functions in that file
20:34   sistpoty        mok0: yes, a local global *g*
20:34   bddebian        hrh
20:34   slangasek       geser: since they're local, they're by definition not accessible to other things linking to this code; so aside from debugging symbols, which we've used 'strip' to wipe out, there's no reason to take up space by keeping around names for these variables
20:35   sistpoty        ok, now let's take a look at the meta-information:
20:35   sistpoty        objdump -x libexample.so
20:36   sistpoty        this will give lots of info. The most interesting stuff can be found with objdump -p libexample.so as well
20:37   bddebian        Who uses anything more than |grep SONAME? ;-)
* slangasek raises his hand
20:37   sistpoty        one thing that's in there is the SONAME, we'll take a look at it later
20:37   sistpoty        the other thing in there is the NEEDED entry (or entries)
20:38   sistpoty        the SONAME denotes the ABI-Verison of the shared object...
20:38   sistpoty        everything with the same SONAME can be considered to be compatible
20:39   geser   if upstream doesn't make a mistake
20:39   bddebian        geser: Never happens ;)
20:39   slangasek       everything with the same SONAME *will be* considered compatible by ld.so, so it darn well better *be* compatible :)
20:39   sistpoty        geser: we'll just see what happens then ;)
20:39   sistpoty        the NEEDED entries denote what libraries this shared object needs. These refer to the SONAME of other libraries
20:40   sistpoty        I guess everyone knows ldd, let's try that one
20:40   sistpoty        ldd libexample.so
20:41   sistpoty        ldd will resolve any shared libraries a binary or a shared object needs
20:42   geser   is linux-vdso.so.1 provided by the kernel?
20:42   sistpoty        so, what's the difference between the NEEDED entries and the output of ldd apart from the pathname?
20:42   slangasek       geser: yes, linux-vdso is a bit of Magic
20:43   warp10  ldd output shows three entries, not just one
20:43   geser   ldd resolves them recursively
* bddebian thinks geser is too smart for this class :)
20:43   sistpoty        exactly what geser said :)
20:43   geser   bddebian: :P
20:43   sistpoty        so you should keep these in mind when dealing with shared objects, as you might want to look at the *direct* dependencies as well
20:44   slangasek       yes; for a real world-example, have a look at ldd /usr/lib/libgnome-2.so.0, vs. objdump -p /usr/lib/libgnome-2.so.0 | grep NEEDED
20:45   slangasek       so we've looked at the output of nm -D on this object, but nm -D has one weakness as a tool which is not at all obvious if you don't already know what's missing
20:46   slangasek       have a look at the output of nm -D libexample.so again
20:46   slangasek       now, compare it to the output of readelf -s libexample.so
20:46   slangasek       who can spot the difference in the names?
20:47   bddebian        The stuff from @@glibc?
20:47   geser   you mean the versioned symbols?
20:47   slangasek       yep - nm -D doesn't show symbol versions
20:47   mok0    cool
20:48   slangasek       so if you want to get at those, you'll want to use objdump -T or readelf -s
20:48   geser   what does the "(2)" in "6: 0000000000000000   144 FUNC    GLOBAL DEFAULT  UND fprintf@GLIBC_2.2.5 (2)" mean?
20:48   slangasek       symbol versions are sometimes important, so you'll want to keep these tools in mind -- sometimes, an app/lib will not know the symbol version it's supposed to use, and that indicates a bug!
20:49   slangasek       geser: I'm not sure, sorry
20:50   slangasek       I'd probably have to look at the readelf source to figure that out, the fields aren't all well-explained in the documentation
20:50   mok0    we can write a dependency resolver from these tools
20:50   slangasek       mok0: indeed, dpkg-shlibdeps uses these tools
20:51   sistpoty        ok, any more questions so far?
20:51   sistpoty        When writing a program, that uses a library, we want to make sure that the API of that library is stable.
20:52   sistpoty        If we want to distribute this program, we also want a stable ABI.
20:52   sistpoty        stable API means that my program compiles against the library and a newer version of it.
20:52   sistpoty        stable ABI means that the symbols stay the same
20:53   mok0    what if there are new symbols?
20:53   sistpoty        mok0: of course... I just wanted to correct myself
20:53   sistpoty        stable ABI means that there won't get any symbols removed, and their meaning is still the same
20:55   mok0    If you write code that corrects a bug, but the interface remains the same?
20:55   sistpoty        mok0: then the ABI and API is stable
20:56   sistpoty        (most likely)
20:56   mok0    ... but I would still want that programs exclusively use the fixed version
20:56   sistpoty        but let's first have some fun... as some newby upstream started to produce a library...
20:56   geser   mok0: if I'm correct you can change the implementation as long as function name, arguments and return type stay the same
20:57   sistpoty        mok0: you can do this (as upstream) only, if the library have some version string or date encoded
20:57   mok0    ok
20:57   sistpoty        mok0: or by advetising it on your homepage *g*
20:57   mok0    you can bump the minor version number
20:58   sistpoty        for the library: yes (I'll come to this part yet)
20:58   chiptoxic       how do you tell if a library has a stable API or ABI is it just by watching the version numbers of that library?
20:58   sistpoty        chiptoxic: I'd like to postpone this question, as I first want to show you an example
20:59   chiptoxic       ok
20:59   sistpoty        so please everyone get <http://www.potyra.de/library_packaging/libmyhello-1.0.1.tar.gz>
20:59   sistpoty        if you've got it, extract it and compile it with
20:59   sistpoty        make
21:00   sistpoty        then install the result (as root) with
21:00   sistpoty        make install
21:00   sistpoty        (no worries, it will only copy files to the /usr/local/ namespace, and make uninstall removes it again)
21:01   sistpoty        everyone got so far? (then me types the commands as well)
21:02   mok0    make PREFIX=. install
21:02   mok0    does it locally
21:03   sistpoty        then you might have trouble upgrading the library
21:04   sistpoty        (or building an application against it, which we'll do as soon as everyone is finished)
* warp10 is done
21:04   sistpoty        ok, next step: of course we want s.th. that uses the library, so get <http://www.potyra.de/library_packaging/hello_prog-1.0.tar.gz>
21:05   sistpoty        extract that somewhere else and also build the binary with "make"
21:06   sistpoty        if you've done that, you should have a cool binary hello_prog, which will print hello. you can even tell it how often it should print hello with the first argument
21:06   sistpoty        ./hello_prog 10
21:07   sistpoty        everyone got so far yet?
21:07   mok0    error while loading shared libraries: libmyhello.so.1: cannot open shared object file: No such file or directory
21:07   frigaut same here
21:07   geser   mok0: sudo ldconfig
21:08   mok0    geser: just doing what the teachers tell me .-)
21:07   warp10  and here too :)
21:07   slangasek       heh, did any of the people having this problem install to /usr/local?
21:08   slangasek       or did you all install to a different prefix?
21:08   james_w slangasek: /usr/local for me
21:08   slangasek       ah, ok
21:08   sistpoty        hehe
21:08   frigaut works now after ldconfig
21:08   sistpoty        of course I forgot to mention ldconfig
21:08   chiptoxic       mine works and i just did what you said and ran ldconfig
21:08   sistpoty        (still got it installed from testing *g*)
21:09   mok0    Yay, it says hello to me 10 times
21:09   kdub    you have to have /usr/local in your /etc/ld.so.conf of course
21:09   geser   sistpoty: "eventually run ldconfig now."
21:09   slangasek       kdub: in /etc/ld.so.conf.d/libc.conf actually... :)
21:09   sistpoty        geser: hehe, what a nice upstream *g*
21:10   geser   yep
21:10   sistpoty        now if it prints hello for everyone, let's remove the *library* again with
21:10   sistpoty        make uninstall (run as root)
21:11   sistpoty        now try again to run the hello_prog.
21:12   sistpoty        any results yet?
21:12   geser   ./hello_prog: error while loading shared libraries: libmyhello.so.1: cannot open shared object file: No such file or directory
21:12   mok0    boom
21:13   sistpoty        great, so (I guess what you expected) it won't work w.o. the library
21:13   sistpoty        now upstream has released a new version of the library... I guess you all want hot new stuff?
21:13   sistpoty        then get it at http://www.potyra.de/library_packaging/libmyhello-1.0.2.tar.gz
21:13   sistpoty        we'll be compiling it again:
21:13   sistpoty        make
21:14   sistpoty        and install it again (run as root)
21:14   sistpoty        make install
21:14   mok0    If you install the library again, it works once more :-)
21:14   sistpoty        hehe
21:14   warp10  I don't think so :)
21:15   geser   ./hello_prog: symbol lookup error: ./hello_prog: undefined symbol: print_hello
21:15   sistpoty        (oh, if you haven't removed installed the new library, you could also try ldd ./hello_prog)
21:16   sistpoty        so what happened?
21:16   sistpoty        now let's try to make use of the knowledge we gathered so far
21:16   mok0    the application attempts to use the new library
21:17   sistpoty        mok0: exactly
21:17   sistpoty        let's take a look at the symbols of both shared objects (old library vs. new library)
21:17   mok0    => ABI has changed
21:17   sistpoty        excellent!
21:18   sistpoty        in the old library, we had the function: T print_hello
21:18   sistpoty        which upstream obviously renamed to just: T hello
21:19   sistpoty        when you start the program, one of the first steps is that undefined symbols will get resolved
* bddebian curses upstream
21:19   sistpoty        hehe
21:20   slangasek       yeah, I would think twice about packaging any libraries from this upstream ;)
21:20   sistpoty        which is done recursively for each shared object that is used
* sistpoty removes copyright notices *g*
21:20   mok0    This is why it is wrong to use the program version number for library versions
21:20   bddebian        heh
21:20   mok0    which a lot of upstream programmers don't understand
21:21   sistpoty        when the symbol lookup can't resolve one symbol.. well you just saw what happens.
21:21   sistpoty        ok, what we've seen now is an incompatibility on symbol level.
21:22   sistpoty        I hope with the tools you've learned so far, you can easily track those down
21:22   sistpoty        -> any removed symbol in a shared library is a problem, *if* the SONAME stays the same
21:23   mok0    so let's change it
21:24   sistpoty        mok0: feel free to change it, I guess you can figure this in the Makefile
21:24   mok0    I set major to 2
21:25   albert23        Wouldn't this be an API change, since print_hello is not defined anymore in hello.h?
21:25   albert23        I wouldn't expect this to compile anymore?
21:25   sistpoty        albert23: exactly: this is an API change as well
21:26   sistpoty        ok, but let's get back to the ABI...
21:27   sistpoty        symbols however cannot represent the signature of functions (i.e. number of parameters, type of parameters)
21:27   sistpoty        this is part of the ABI as well
21:27   sistpoty        if we want to look if this changed, we'll need to dig a little bit deeper
21:27   mok0    Now the program works again, because it finds the first version of the library
21:27   mok0    (I changed MAJOR to 2)
21:28   sistpoty        mok0: hehe, so you didn't unstall the old library?
21:28   chiptoxic       im not a programmer but is there a way you could put an alias in the code say print_hello is really hello so you dont break the api? or would that just be messy
21:28   mok0    sistpoty: I still have 1.0.1 but I removed 1.0.2
21:28   mok0    The new one is 2.0.1
21:29   sistpoty        chiptoxic: you could for example abuse the preprocessor: "define print_hello hello"
21:29   mok0    My experiment shows you can have several versions of the library present
21:29   sistpoty        er... #define print_hello hello
21:29   chiptoxic       ah cool, thanks
21:29   sistpoty        ok, back to the topic...
21:30   sistpoty        you can look at the debug information with gdb, so let's run
21:30   sistpoty        gdb libmyhello.so
21:30   slangasek       mok0: exactly, and this is a key reason why we always use the full soname for the names of our library packages
21:30   geser   mok0: and now try the same on windows :)
21:30   sistpoty        inside gdb, you can look at the signatures of functions with
21:30   sistpoty        info functions
21:31   mok0    slangasek: you mean the major soname ?
21:31   slangasek       mok0: the "soname" is the value returned by objdump -p $lib | grep SONAME
21:31   mok0    slangasek: ok
21:31   slangasek       and set (hopefully) using the -Wl,-soname option at build time
21:32   sistpoty        mok0: the Makefile just puts $(MAJOR) and a name in the soname, but the real soname is in SONAME
21:32   slangasek       lintian knows the standard way to map sonames to package names, and warns you if your package is named differently.  but of course, we have to take care that upstream is handling their ABI right in the first place!
21:32   sistpoty        inside gdb, you can also display the types of variables with
21:32   sistpoty        info variables
=== \sh is now known as \sh_away
21:33   sistpoty        (which is uninteresting for libmyhello.so though, since it doesn't define a global variable by itself)
21:33   sistpoty        everyone looked at the output of "info functions"?
21:34   mok0    just one: hello(void)
21:34   sistpoty        you should see s.th. like "void hello(void);" or "void print_hello(void);" there (depending which version you chose)
21:34   sistpoty        yes, so upstream is lazy as well *g*
* bddebian files upstream bug
21:35   sistpoty        now let's strip the shared object (which wasn't done during the build process):
21:35   sistpoty        strip libmyhello.so
21:35   sistpoty        (outside gdb again)
21:36   sistpoty        and then call gdb libmyhello.so again and look at the functions again
21:36   mok0    :-(
21:36   sistpoty        as you can see, these are all gone.
21:36   geser   mok0: install the -dbgsym package of it :)
21:37   mok0    heh
21:37   sistpoty        that's because these were part of the debug symbols, which have been stripped away
21:37   sistpoty        ok, so if we want to find out about the ABI-part of the signatures, it's a little bit tougher, but still doable
21:39   sistpoty        if a signature is changed (number of arguments and type of a function), it can lead to different results. Most probably it will segfault, since the library and the application no longer agree on what data they exchange)
21:39   sistpoty        an exception to this rule is C++
21:40   sistpoty        in C++ the signature is encoded in the name of a symbol
21:40   sistpoty        so if you change an argument, add an argument or remove one, you get a different symbol name
21:41   sistpoty        (side note: you can use nm -C to demangle these names back to a human readable signature)
21:41   slangasek       or c++filt
21:41   sistpoty        any questions so far?
21:42   sistpoty        chiptoxic: did that answer your question from some time ago?
21:42   slangasek       other side note, checking for ABI changes of this kind by hand is hard and tedious; there is a tool in universe that tries to automate this checking, called icheck
21:42   chiptoxic       yea, thanks
21:42   mok0    slangasek: icheck, what package?
21:43   sistpoty        slangasek: ah, nice :)
21:43   sistpoty        mok0: icheck
21:43   slangasek       if you're maintaining a library and want to be careful because your upstream isn't, icheck will let you generate a "manifest" from the old and new -dev packages, and compare them to see if the signatures have changed
* chiptoxic cant believe how much im learning :)
21:43   james_w slangasek: how does that compare to check-symbols from the ubuntu-dev-tools package?
* mok0 install icheck
21:43   slangasek       james_w: icheck examines headers, not symbols
21:44   slangasek       james_w: so it's specifically for the case where you're trying to detect the kinds of ABI changes that /don't/ show up in the symbol table
* bddebian can't keep up being at work :'-(
21:45   slangasek       (doh, check-symbols uses nm -D?  Baaad, what did we learn was the weakness of nm -D? :-)
21:45   sistpoty        slangasek: since you mentioned ABI changes not showing up in the symbol table, want to say a few words to these?
21:46   slangasek       probably, let me see where I put those words
21:47   slangasek       right - does everyone here understand why changing the arguments or return type of a function is an ABI change?
21:47   mok0    yes
21:47   slangasek       (if everyone knows, there's no sense in me spending time explaining it - but if you don't know, please speak up :)
21:48   warp10  y
21:48   mok0    Well renaming a global variable would show up
21:49   slangasek       two people know who care, everyone else is asleep or doesn't want to talk to me ;)
21:49   slangasek       sistpoty: back to you
21:49   slangasek       :)
21:49   mok0    I just tried icheck on the 2 library versions,
21:50   mok0    but I couldn't get it to work
21:50   sistpoty        ok
21:50   mok0    ... "Failed to parse libmyhello.so
21:50   sistpoty        mok0: haven't used it myself yet
21:51   mok0    sistpoty: :-)
21:51   sistpoty        what we should have learned so far is that if the ABI breaks, you'll need to bump the SONAME
21:51   slangasek       mok0: you point it at the headers, not at the library
21:51   mok0    Ah!
21:51   sistpoty        however there is yet another case, we didn't consider
21:51   slangasek       that's the whole point - the library doesn't have the info you need about the function sigs :)
21:52   sistpoty        upstream might put additional functionality in the library
21:52   sistpoty        and keep the ABI downwards compatible
21:53   sistpoty        in that case, the SONAME would (and should) stay the same, because no program ceases to work with the newer library
21:53   sistpoty        however this has one caveat: people who use this library might make use of the new features
21:53   mok0    ... but new programs that use the new functionality?
21:54   sistpoty        right, so these programs wouldn't work with the old library but only with the new one (even though the SONAME is the same)
21:54   sistpoty        for this, there the SONAME is not sufficient alone.
21:55   sistpoty        slangasek: told me that historically the minor version was incremented for this
21:55   sistpoty        -:
* mok0 also read that
21:55   sistpoty        but that won't work if you copy binaries from one system to another (as you don't know about the libraries there)
21:56   sistpoty        ok, this pretty much was part 1 -- shared objects
21:56   sistpoty        any questions so far?
21:57   mok0    Well, what is the answer to the last problem?
21:57   sistpoty        mok0: we'll see that in part 2 :)
21:57   mok0    Arrghh
21:57   warp10  :)
21:57   mok0    It stops at the most exiting part
21:58   mok0    typical of these mini-series :-)

20:06   sistpoty        so who's around for part 2 of the library packaging session?
20:06   albert23        hello sistpoty
20:06   sistpoty        hi albert23
20:07   sistpoty        first off, there is an excellent guide about debian library packaging. We'll use this one as a reference for this session:
20:07   sistpoty        http://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html
20:09   nixternal       thanks sistpoty for the wakeup call :)
20:09   sistpoty        hi nixternal :)
20:09   nixternal       well howdy
20:09   sistpoty        ok, in part 1, we hopefully learned all the basic stuff about shared objects.
20:10   ScottK  You wouldn't have a link to the log for that would you?
20:10   nixternal       I was just gonna ask, I have a lib I gotta package up :)
20:10   albert23        http://irclogs.ubuntu.com/2008/01/17/%23ubuntu-classroom.html
20:10   sistpoty        ScottK: sorry.. no idea if this is publically logged somewhere
20:10   sistpoty        thanks albert23
20:11   ScottK  albert23: Thanks
20:11   sistpoty        now in part 2, we'll be packaging a library by example, and will discuss a few of the caveats
20:12   sistpoty        since I'm lazy, let's take libmyhello again from part 1
20:12   sistpoty        http://www.potyra.de/library_packaging/libmyhello-1.0.1.tar.gz
20:13   sistpoty        everyone got it?
20:13   warp10  y
20:13   hellboy195      y
20:13   albert23        yep
20:13   sistpoty        ok, let's start with debianization... unpack it, rename the tarball so that we've got a valid .orig.tar.gz
20:14   sistpoty        let's take a look, we'll always need a few files in there, let's start with debian/copyright
20:16   sistpoty        put in the usual stuff there (this is not really important for the technical bits of library packaging)
20:17   sistpoty        what else do we need... I guess we'll be using debhelper, so let's create a compat in debian/
20:17   sistpoty        everyone got it so far?
20:17   hellboy195      y
20:17   warp10  y
20:18   albert23        y
20:18   sistpoty        ok, now let's start with the interesting bits
20:18   sistpoty        the control-file
20:19   sistpoty        First off comes the usual stuff (I'll pastebin this)
20:19   sistpoty        hm... pastebin is kinda slow today :(
20:20   sistpoty        ok, I'll paste it right here... pastebin doesn't like me *g*
20:21   sistpoty        Source: libmyhello
20:21   sistpoty        Section: libdevel
20:21   sistpoty        Priority: optional
20:21   sistpoty        Maintainer: Your Name <youraddress@bla.org>
20:21   sistpoty        Build-Depends: debhelper (>> 5.0.0)
20:21   sistpoty        Standards-Version: 3.7.2.0
20:22   sistpoty        for a library, we'll always end up with two packages
20:22   sistpoty        two binary packages even
20:22   sistpoty        one package contains the shared object(s)
20:23   sistpoty        and another one, the -dev package will contain everything you need to develop applications with the shared object
20:24   sistpoty        i.e. header-files, eventually a statically linked library, eventually pkconfig files (which specify what to pass to the compiler/linker to link against this library)
20:25   sistpoty        so let's create an entry for the -dev package
20:25   sistpoty        Package: libmyhello-dev
20:25   sistpoty        Architecture: any
20:25   sistpoty        Description: myhello library headers
20:25   sistpoty        This package contains the development headers
20:25   sistpoty        for libmyhello.
* ScottK notes sistpoty needs to update to the current standards version...
20:26   sistpoty        ScottK: hehe, I'm not up to date
20:27   sistpoty        the package containing the shared object will need to be named in a special way:
20:28   sistpoty        it must be related to the soname.
20:28   sistpoty        the library packaging guide I mentioned contains the rules how it should be named
20:29   sistpoty        chapter 5, section 3
20:29   sistpoty        remember the soname of libmyhello yet?
20:30   sistpoty        anyone?
20:30   warp10  mmm... libmyhello.so.1, maybe?
20:30   sistpoty        warp10: exactly
20:31   sistpoty        if you wouldn't have remembered, you could of course always use objdump -p <shared object> to find out
20:33   sistpoty        so in this case, the name of the library would be libmyhello1
20:33   sistpoty        if you've got a library, that contains more than one shared object
20:33   sistpoty        you'll need to create a binary package for each of these
20:33   sistpoty        let's add an entry for the binary package
20:34   sistpoty        Package: libmyhello1
20:34   sistpoty        Architecture: any
20:34   sistpoty        Depends: ${shlibs:Depends}
20:34   sistpoty        Description: myhello library
20:34   sistpoty        This package contains the hello library.
20:35   sistpoty        ok. anyone got a clue, why the name of this binary must be related to the SONAME?
20:35   sistpoty        this binary package...
20:36   sistpoty        anoyne`
20:36   sistpoty        +?
20:36   albert23        we want to be able to install multiple versions of the library, when ABI changes
20:36   sistpoty        albert23: excellent
20:37   sistpoty        so does anyone have an idea why this could be needed?
20:38   warp10  sistpoty: maybe different packages may need different versions of the library?
20:38   sistpoty        warp10: this would be one argument, but not the main one.
20:38   sistpoty        let's just consider that a very essential library (maybe libc) will get updated
20:39   sistpoty        currently, I've got libc6 installed, and I guess almost all packages I've got installed will depend on libc6
20:39   sistpoty        suddenly, a new libc7 appears
20:40   sistpoty        if it couldn't be installed side-by-side to libc6, that would mean that I had to remove *all* installed programs
20:40   sistpoty        or that all programs would immediately break if libc6 were removed by force
20:40   sistpoty        that's pretty bad, hm?
20:41   sistpoty        (I tried this once when I didn't have a clue... wasn't so funny back then *g*)
20:41   warp10  :)
20:41   sistpoty        any questions so far?
20:42   warp10  I have one
20:42   warp10  Why the -dev package doesn't need any Depends: field?
20:42   sistpoty        warp10: excellent question... it needs one :)
20:43   sistpoty        so, as I wrote earlier, the -dev package is what you install if you want to build programs using this library
20:44   sistpoty        so we need to make it depend at least on the library package
20:44   sistpoty        let's add the depends line there
20:45   sistpoty        the header files in question should always match what's in the shared object
20:45   sistpoty        hence we'll need to restrict the version for this dependency
20:46   sistpoty        Depends: libmyhello1 (= ${binary:Version})
20:47   sistpoty        however often you'll need additional libraries to compile a program with a specific library
20:47   sistpoty        so you'll need to add the corresponding -dev packages there as well.
20:48   sistpoty        a good indication should give you the NEEDED entries of the shared object
20:49   sistpoty        so you'll just need to find the corresponding -dev package that contains these shared objects
20:50   sistpoty        so, what do we need here?
20:51   albert23        NEEDED      libc.so.6
20:51   sistpoty        great.
20:52   sistpoty        so let's find out in which package this is (we'll see a better method later)
20:52   sistpoty        just use dpkg -S libc.so.6
20:53   sistpoty        hooray, it's libc6 :)... so we'll want libc6-dev
20:54   sistpoty        (libc6-dev is often ommitted, since build-essential already depends on it. however it's not wrong to add it)
20:55   sistpoty        as you might have noticed, in the case of libc6, the -dev package is not named like the source package
20:55   sistpoty        but instead it is named like the binary package + -dev
20:56   sistpoty        this is useful, if you want to have more than just one version of the library in the archive
20:56   sistpoty        (e.g. because one application needs the old one, and won't build with the new version)
20:56   sistpoty        then you'll need to be able to select which -dev package you want
20:57   ScottK  There's a very recent thread on adding libc6 on debian-devel that came down pretty strongly in favor of adding it.
20:57   sistpoty        if you do this, you should have the -dev package(s) of the different versoin Provide and Conflict to libmyhello-dev package
20:58   sistpoty        ScottK: ah, haven't read that one yet
20:58   sistpoty        (or maybe I have but forgot *g*)
20:59   sistpoty        ok, does everyone have the control file so far?
20:59   albert23        y
20:59   warp10  y
21:00   sistpoty        hehe, mine is still missing the depends line *g*
21:01   sistpoty        ok
21:01   sistpoty        what else do we need... a changelog file. just create one with dch
21:03   sistpoty        dch --create
21:03   sistpoty        (needed to look this up, thought it always was -c)
21:04   sistpoty        everyone got it filled in?
21:04   warp10  y
21:05   albert23        y
21:05   sistpoty        ok, then let's start with the rules file
21:06   sistpoty        let's see what targets we need to adhere to the policy: clean, build, binary, binary-arch, binary-indep
21:06   sistpoty        did I miss one?
21:07   warp10  install?
21:07   ScottK  configure?
21:07   slangasek       not required by policy :)
21:07   sistpoty        yep
21:07   warp10  configure should be optional, right?
21:07   sistpoty        hi slangasek btw.
* slangasek waves
21:08   warp10  install is optional too?
21:08   sistpoty        warp10: libmyhello fortunately doesn't need to be configured, as it doesn't use autotools
21:08   sistpoty        warp10: yes
21:09   sistpoty        let's start with clean, should be easiest
21:09   sistpoty        since we'll use debhelper, we'll first call dh_clean
21:09   sistpoty        and then: make clean
21:09   sistpoty        to cleanup libmyhello stuff
21:09   sistpoty        or, if you prefer make distclean
21:10   sistpoty        (which won't make a difference *here*)
21:12   sistpoty        then let's get a shot at the build rule
21:12   james_w sistpoty: dh_testroot to start clean as well?
21:13   sistpoty        james_w: right, makes sense
21:14   sistpoty        ok, build should be easy as well.
21:14   sistpoty        just a call to make
21:14   sistpoty        any questions so far?
21:15   warp10  n
21:15   albert23        n
21:15   sistpoty        ok, then let's go to the binary target
21:16   sistpoty        since this is an arch:any package, we'll use the binary-arch target for the job, and have the binary target depend on it
21:17   sistpoty        let's add some debhelper stuff
21:18   sistpoty        dh_testdir
21:18   sistpoty        dh_testroot
21:18   sistpoty        dh_installchangelogs
21:18   sistpoty        dh_installdocs
21:19   sistpoty        dh_install
21:19   sistpoty        dh_strip
21:19   sistpoty        dh_compress
21:19   sistpoty        dh_fixperms
21:20   sistpoty        and now an interesting one: dh_makeshlibs
21:20   sistpoty        (we'll have a look at it in a few minutes)
21:20   sistpoty        dh_shlibdeps
21:20   sistpoty        dh_installdeb
21:20   sistpoty        dh_gencontrol
21:20   sistpoty        dh_md5sums
21:20   sistpoty        dh_builddeb
21:21   sistpoty        since we'll need to build s.th. before we can install it, we'll add a dependency to build for binary-arch
21:22   sistpoty        (a nicer way is to use stamps, but that's a different topic)
21:22   sistpoty        now what does dh_makeshlibs do...
21:23   sistpoty        as you may have noticed, whenever you put ${shlibs:Depends} in the control file, it will get resolved to the library packages during build
21:23   sistpoty        this is done via the shlibs-mechanism
21:23   sistpoty        if you've been here in part 1, I guess you could build one yourself
21:24   sistpoty        if you look at /var/lib/dpkg/info, you'll find a number of shlibs-files
21:25   sistpoty        let's take a look at one
21:26   sistpoty        the format is: library-name soname-version dependencies
21:27   sistpoty        let's look at /usr/lib/dpkg/info/libgnome2-0.shlibs
21:28   sistpoty        compare this to the SONAME of /usr/lib/libgnome-2.so.0
21:28   sistpoty        this explains the first two fields
21:29   sistpoty        the third one is the package where this shared object can be found in
21:29   sistpoty        the job of dh_makeshlibs is to generate such a file for us
21:29   warp10  cool...
21:31   sistpoty        and at this point, we can try to answer mok0's question of part 1, what to do if upstream released a new abi compatible version, but programs use this one
21:31   sistpoty        since the shlibs file can specify a version string (just look at the gnome shlibs file again)
21:31   sistpoty        we can request to have a minimum version of the library
21:32   sistpoty        this will get replaced for all packages that use ${shlibs:Depends} and build against our library
21:33   sistpoty        so we only need to tell dh_makeshlibs, that a new feature landed in the library, so that it will set version field in the shlibs file right
21:33   sistpoty        you can do this with dh_makeshlibs -V<version>
21:33   sistpoty        now if anyone has installed an older version of the library, and wants to use the program that needs the new cool features
21:34   sistpoty        he will have to upgrade the library via apt
21:34   sistpoty        any questions so far?
21:34   albert23        How can we know which library version was the first providing a specific ABI version?
21:35   albert23        Isn't there a risk we set version to high in dh_makeshlibs -V<version>?
21:35   sistpoty        albert23: pretty much through the same means that we can find out if the ABI is broken
21:35   albert23        ok
21:36   sistpoty        albert23: there is no real risk in Ubuntu to set the version to high
21:36   sistpoty        however setting the version not high enough can lead to problems
21:36   albert23        It would make backports more difficult?
21:37   sistpoty        not too sure... I don't think so
21:37   sistpoty        a backport gets compiled to whatever is in gutsy or feisty
21:37   sistpoty        so it will get built against the feisty/gutsy version of the library, which has a different version requirement
21:37   ScottK  So if you set it correctly and there isn't a sufficient ABI available, the backport build will, correctly, fail.
21:38   sistpoty        well... it doesn't really relate to building, but rather to installin
21:38   sistpoty        +g
21:39   sistpoty        in order to build against a library, the version (and ABI) is irrelevant.
21:39   sistpoty        what matters to build against it is the API
21:39   sistpoty        the version rather enforces that users *need* to upgrade the library to a minimum ABI version
21:39   ScottK  I guess I meant API then.
21:40   sistpoty        so if the API changed, the field won't really be much of a problem, since this means a new ABI in almost all cases
21:40   sistpoty        and hence a new SONAME, and a different library package
21:41   sistpoty        this version is only relevant if the ABI/API stay the same, but new features got added
21:41   ScottK  Ah.  RIght.
21:41   sistpoty        any further questions?
21:42   warp10  n
21:42   albert23        n
21:42   sistpoty        ok, then let's try to fix the rules file and finish the package, shall we?
21:43   sistpoty        since we use dh_install, we still need to install the files into debian/tmp/
21:43   sistpoty        let's use PREFIX=$(pwd)/debian/tmp make install for this
21:44   sistpoty        if you want, you can put this right at the beginning of binary-arch
21:44   sistpoty        or you could add an install rule and have binary-arch depend on it... whatever you prefer
21:46   sistpoty        now we'll need to sort which files go into which package
21:47   sistpoty        so we want libmyhello1.install and libmyhello-dev.install
21:47   sistpoty        the header files will go into the -dev package
21:49   sistpoty        bah... back a few lines... (recalled the Makefile wrongly)... PREFIX should of course be $(pwd)/debian/tmp/usr
21:49   sistpoty        I guess I should have added DESTDIR or s.th. to libmyhello's makefile
21:50   sistpoty        in case we'd have a statically linked library (libmyhello.a), that would go into the -dev package as well.
21:51   sistpoty        (only useful if you want to link s.th. statically against it)
21:52   sistpoty        what else do we need in the -dev pacakge
21:52   sistpoty        the .so symlink should go in there as well
21:53   sistpoty        btw: the .so symlink is what the linker will look for if you give the -l parameter
21:53   sistpoty        more specifically, if you'd call -lmyhello, it would prepand a lib and add an .so and look for this file (libmyhello.so)
21:54   sistpoty        in case there are any other files for development purposes (pkconfig files for example), these should also go in the -dev package
21:54   sistpoty        luckily we don't have these
21:56   sistpoty        in the library package should be the shared object (of course) and the symlink that's named like the soname
21:56   sistpoty        while the symlink itself would also be recreated by ldconfig
21:57   sistpoty        including it in the library package has the advantage that it will get deleted if the library is removed
21:57   sistpoty        because it's a file of the package
21:58   sistpoty        everyone got the .install files?
21:58   albert23        y
21:58   warp10  y
21:59   sistpoty        ok, let's give it a shot, and see if it works
21:59   sistpoty        just use
21:59   sistpoty        fakeroot make -f debian/rules binary
22:00   sistpoty        to fix the errors
22:00   sistpoty        (like my install files, that where wrong *g*)
22:00   albert23        mine too
22:00   warp10  argh... :-S
22:01   sistpoty        did you fix the PREFIX=... make install?
22:01   albert23        y
22:01   warp10  sistpoty: just a wrong install for me too :)
22:01   sistpoty        warp10: hehe
22:04   albert23        I think $(pwd) did not work
22:04   albert23        make tried to install in /debian
22:05   sistpoty        it should install into debian/tmp/usr/
22:05   albert23        I got this: PREFIX=/debian/tmp/usr make install
22:06   sistpoty        heh, I've got the same which doesn't work as well. but the answer is one line above
22:06   sistpoty        PREFIX=debian/tmp/usr make install
22:06   sistpoty        those nasty slashes *g*
22:06   warp10  grrrr
22:06   albert23        Yep, just removed the pwd part, that works
22:09   sistpoty        hm... I guess I'll need to read the make manual again, I was so sure that $(PWD) exists (or $(pwd))
22:09   albert23        $(CURDIR) ?
22:09   slangasek       $(PWD) exists only if it's set in the environment by the invoking shell
22:10   slangasek       $(shell pwd) execs out to call the pwd command, so is rather inefficient
22:10   slangasek       $(CURDIR) is the make builtin variable you should use
22:11   sistpoty        thanks... I guess I confused this with $(pwd), which I often type in the shell *g*
22:11   slangasek       :)
22:11   sistpoty        so does everyone's build succeed now?
22:12   albert23        yep, package looks good
22:12   warp10  looks fine!
22:12   sistpoty        damn, only mine doesn't build yet *g*
22:13   sistpoty        ok, so you've got a source package and a binary
22:13   sistpoty        let's run lintian at the binary packages. it will take a second look that we've got the files sorted out correctly
22:14   albert23        clean
22:15   sistpoty        excellent!
22:15   sistpoty        any questions so far?
22:16   albert23        no, that's pretty clear
22:17   warp10  it's fine. Just wondering if it would be so easy in the real-world packaging :)
22:18   albert23        I guess that strongly depends on upstream
22:18   sistpoty        ok, so now what would you do if the ABI of a library changed, but the SONAME didn't?
* slangasek smiles
22:19   albert23        complain at upstream
22:19   sistpoty        albert23: great, that should always be the first choice
22:19   sistpoty        can you imagine, while it's a bad idea, if you bump the SONAME on your own?
22:20   sistpoty        (as a packager)?
22:20   albert23        could be risky, as it might conflict with new upstream versions?
22:21   sistpoty        that's one argument
22:22   sistpoty        but you could append a debian or ubuntu somewhere and change the build scripts, so you could circumvent this
22:22   sistpoty        any other arguments against this?
22:22   slangasek       it leads to binary incompatibility with other distros
22:22   slangasek       but that shouldn't outweigh the problems caused by binary incompatibility with previous releases
22:23   sistpoty        also, it leads to binary incompatibility for binary only software
22:23   sistpoty        (just imaging to change the SONAME of libc, and look at what breaks in multiverse)
22:24   slangasek       well, a workaround for such binary-only software would be to instruct users to create their own local symlinks for the wrong soname
22:25   sistpoty        heh
22:25   sistpoty        ok, what other interesting points do we have...
22:25   slangasek       like I said, the alternative is that you have to break all the /free/ software using the old version on upgrade :)
22:25   sistpoty        yeah, it's either bad or worse *g*
22:26   sistpoty        ok, what should you do when you upgrade a library?
22:26   slangasek       by "upgrade", you mean "package a new version"?
22:26   sistpoty        yes
22:27   sistpoty        we already learned that we should look out for ABI compatibility, so let's look at the case where we *do* have a new SONAME
22:27   sistpoty        and hence a new library package
22:28   sistpoty        anyone any ideas?
22:28   slangasek       grep -r $oldsover debian/ :)
22:28   albert23        I think we are going to have a problem with the -dev package
22:29   sistpoty        albert23: what do you think could get problematic?
22:29   albert23        packages requiring the old ABI will get the new ABI when we would want to rebuild?
22:30   sistpoty        exactly (but it's API in this case... compiling *P*rograms->aPi, binaries->aBi)
22:31   sistpoty        so you should check which packages actually build-depend on the -dev package.
22:31   sistpoty        if there is a large number, you could do local test-builds and see which fail
22:32   sistpoty        in case you cannot live yet with the new version alone, you could for example make a different source package and hence have two version of the library in the archive
22:33   sistpoty        one thing you might stumble over when dealing with libraries is a lintian warning about rpath
22:34   sistpoty        when you link against a library, that has an rpath set, it will tell the loader to look for the shared object only in that very directory
22:34   sistpoty        so you can specify -Wl,-rpath=/somewhere and link hello_prog against it
22:35   sistpoty        then the loader will look for libmyhello only in /somewhere when hello_prog is started
22:35   sistpoty        can anyone imagine, why using rpath for a library is a bad idea?
22:38   albert23        well, software would break if for some reason the library is moved to another directory....
22:38   sistpoty        exactly
22:38   sistpoty        and one use-case for moving a library is for example the ia32-libs package on amd64
22:39   sistpoty        ok, any questions?
22:39   slangasek       (and multiarch! whoo!)
22:40   slangasek       albert23, sistpoty: I would note that the general case is that we don't want to continue supporting an old API for a library, even if it means some work has to be done to get the applications ported
22:40   slangasek       so it's /usually/ better to have the new library version take over the source package name and the -dev package name, and only reintroduce an oldlibs package as a last resort
22:41   sistpoty        thanks for clarifying
22:41   albert23        Well, one question: what would be the best place to look for bugs / qa issues to practice what we learned?
22:42   sistpoty        hm... that's a good question indeed
22:42   sistpoty        well, one thing is that you could review library packages on REVU
22:43   sistpoty        or get your hands dirty and package a library
22:44   sistpoty        ok, I guess we're through with the session... slangasek: anything important I forgot or you'd like to add?
22:45   slangasek       sistpoty: I only had half an eye on the channel today; I guess you didn't talk about techniques maintainers can use to automatically detect ABI breakage when moving to a new upstream version?
22:45   sistpoty        slangasek: nope
22:46   slangasek       the dpkg in hardy includes the new dpkg-gensymbols code developed by Raphaël Hertzog in Debian; it basically creates a manifest, declaring which symbols were added in which version of the package
22:46   slangasek       the /side/ benefit of this is that if you already know which symbols are supposed to be present, you know if any have gone missing
22:47   slangasek       so http://qa.debian.org/cgi-bin/mole/seedsymbols is a site that includes automatically generated symbol manifests for all libs in the archive
22:47   slangasek       (the Debian archive, that is; most will be applicable to Ubuntu too, obviously)
22:48   slangasek       http://wiki.debian.org/UsingSymbolsFiles is a general description of this
22:48   slangasek       I still need to use this on my own packages, so I can only give you theoretical insight into this stuff, not much practical :)
22:49   slangasek       and everyone's eyes glaze over again ;)
22:50   albert23        Yeah, I guess I will first do some attempts to do the work manually.
22:50   albert23        I have bookmarked those links for later use...
22:51   slangasek       one example of this in practice is the zlib1g package
22:51   slangasek       which the maintainer, Mark Brown, blogged about as well (syndicated on planet.d.o)
22:52   sistpoty        heh, I skipped this post, and wanted to read it later (which I then forgot *g*)
22:52   sistpoty        ok, anything else?
22:53   sistpoty        then thanks everyone for coming :)

This log has been lightly edited for clarity.

MOTU/School/LibraryPackaging (last edited 2008-08-06 16:33:17 by localhost)