Ubuntu Wiki

Dev Week -- Library Packaging - Part 1 -- Stefan Potyra -- Fri, Feb 22

(01:59:39 PM) dholbach: next up we have Stefan sistpoty Potyra - a great guy, MOTU for quite some time and somebody who always manages to make time for you
(01:59:49 PM) db-keen left the room ("Ex-Chat").
(01:59:51 PM) ***sistpoty bows
(02:00:04 PM) sistpoty: thanks for the introduction dholbach :)
(02:00:06 PM) jpatrick: RainCT: /usr/share/applications/kde/ or kde4?
(02:00:07 PM) term-inal: hello
(02:00:10 PM) dholbach: He's going to give you a two hours talk about Library Packaging
(02:00:21 PM) RainCT: jpatrick: both
(02:00:23 PM) dholbach: so keep your favourite drink handy and enjoy the show!
(02:00:25 PM) ***dholbach hugs sistpoty
(02:00:28 PM) phoenix24: Hello sistpoty !!
(02:00:30 PM) ***sistpoty hugs dholbach
(02:00:41 PM) sistpoty: at least I hope, we can do the session in two hours
(02:00:52 PM) sistpoty: (last time, it was much longer, but I try to be short ;)
(02:01:05 PM) sistpoty: ok, so who's around for library packaging?
(02:01:12 PM) sistpoty: raise your hands ;)
(02:01:18 PM) daishujin: +
(02:01:19 PM) phoenix24: raises!
(02:01:23 PM) ***secretlondo is lurking as this will be too hard
(02:01:31 PM) AstralJava: o/
(02:01:36 PM) snewland: same here, lurking
(02:01:38 PM) \sh: sistpoty: renewing my knowledge so let's do it
(02:01:46 PM) dholbach: +1 :)
(02:01:50 PM) ***daishujin hopes he understands some of this session
(02:01:51 PM) ***polopolo is here to learn!
(02:01:54 PM) sistpoty: ok, first off... this session will be a two part session
(02:02:06 PM) barcc: +0.5
(02:02:07 PM) sistpoty: in the first part, we'll learn the grey theory with some practical examples
(02:02:19 PM) sistpoty: -> everything you don't know about symbols and always wanted to ask ;)
(02:02:36 PM) sistpoty: in the second part, we'll take a look at a practical example, together with a few more hints
(02:03:19 PM) sistpoty: for the first part, I guess all you need is readelf, gdb, nm, gcc, objdump (I guess build-essential is enough to have installed for these)
(02:03:44 PM) sistpoty: in the first part, we'll take a close look at shared objects...
(02:03:51 PM) sistpoty: shared objects can be used in two ways:
(02:03:59 PM) sistpoty: 1) as a plugin mechanism (to be loaded with dlopen)
(02:04:05 PM) sistpoty: and 2) as shared libraries
(02:04:15 PM) sistpoty: we'll focus solely on 2) here
(02:04:36 PM) sistpoty: an ELF shared object (.so) is a collection of symbols together with some meta-information
(02:05:00 PM) sistpoty: a symbol denotes any named entity in c (or c++), e.g. a function call or a (global) variable
(02:05:28 PM) sistpoty: one of the meta-information about a symbol in a shared object is the section it resides in
(02:05:50 PM) sistpoty: e.g. if it has a default value, can be executed... we'll soon look at this
(02:05:59 PM) term-inal left the room.
(02:06:15 PM) sistpoty: actually now... so everyone get http://www.potyra.de/library_packaging/example.c
(02:06:25 PM) polopolo left the room.
(02:06:34 PM) sistpoty: oh, if there are any questions, don't hesitate to ask :)
(02:06:41 PM) sistpoty: everyone got that file?
(02:06:54 PM) phoenix24: yep
(02:07:00 PM) sistpoty: then let's compile it:
(02:07:01 PM) snewland: yes
(02:07:02 PM) sistpoty: gcc -c example.c -o example.o
(02:07:37 PM) sistpoty: now let's take a look at the (not yet shared) object file...
(02:07:44 PM) sistpoty: nm example.o
(02:08:19 PM) sistpoty: what you see there, are the symbols in the object file
(02:08:31 PM) sistpoty: the rightmost thing is the name
(02:08:43 PM) sistpoty: the letter before is the type of the symbol
(02:09:05 PM) sistpoty: upper case letters denote, that the symbol is visible outside of the object file as well
(02:09:19 PM) sistpoty: while a lower case letter means that it is local to the symbol
(02:09:52 PM) sistpoty: hence, e.g. extern_global could be used by a different c-file while static_global could be only used from the c-file we just compiled
(02:10:19 PM) sistpoty: short overview of the types, and what they mean:
(02:10:22 PM) sistpoty: t -> text section (may be mapped readonly), executable
(02:10:32 PM) sistpoty: d -> initialized data (statics, initialized globals)
(02:10:40 PM) sistpoty: c -> uninitialized data (uninitialized globals)
(02:10:47 PM) sistpoty: r -> read only data (const variables), not necessarily read only though.
(02:11:05 PM) sistpoty: u -> undefined. We use a symbol from elsewhere, which will get resolved later by the loader
(02:11:17 PM) sistpoty: you can find these in the manpage of nm for reference
(02:11:36 PM) sistpoty: now, let's compare this with the c-code
(02:12:19 PM) sistpoty: for example the symbol "extern_global" is defined as "int extern_global;"
(02:12:54 PM) sistpoty: it's not initialized with a value, hence the type nm spits out is "c"
(02:13:10 PM) sistpoty: and since it can be used from other c-files, it's an upper case letter
(02:13:16 PM) sistpoty: any questions so far?
(02:13:29 PM) snewland: surprisingly clear so far
(02:13:51 PM) sistpoty: ok, let's build a shared object from example.c
(02:14:04 PM) AstralJava: Just a clarification that you probably meant "local to the module", instead of *symbol.
(02:14:37 PM) AstralJava: Err... that was a question, sorry. :)
(02:14:49 PM) sistpoty: *looking* (I pasted some stuff *g*)
(02:14:57 PM) AstralJava: Gotcha. :)
(02:15:29 PM) sistpoty: AstralJava: can you give me the line? ;)
(02:15:37 PM) snewland: sistpoty: while a lower case letter means that it is local to the symbol
(02:15:49 PM) AstralJava: Yes, that.
(02:16:16 PM) sistpoty: ah... right... of course local to the object file (or c-file that gcc translate to an object file)
(02:16:23 PM) sistpoty: <phoenix24> sistpoty: QUESTION : Why is there no symbol information for "int local_var" (local_function)?
(02:16:41 PM) sistpoty: good question, phoenix24: anyone got the answer for him?
(02:16:57 PM) phoenix24: Coz its a local variable ?
(02:17:07 PM) snewland: those would be taken from the local system?
(02:17:15 PM) sistpoty: phoenix24: exactly
(02:17:19 PM) phoenix24: but, static_local_var is static.
(02:17:26 PM) sistpoty: local variables inside functions reside on the stack
(02:17:53 PM) sistpoty: these will get put on the stack, once the function is called, and will get removed when the function returns
(02:17:54 PM) phoenix24: thanks!
(02:18:05 PM) sistpoty: hence these are not part of the object file
(02:18:37 PM) sistpoty: however static local variables will be in the data section... because they keep their value in the next funciton call
(02:18:40 PM) sistpoty: -> part of the object file
(02:18:51 PM) sistpoty: ok, now let's build a shared lib, shall we?
(02:18:59 PM) phoenix24: yep!
(02:19:07 PM) sistpoty: gcc -Wl,-z,defs -Wl,-soname,libexample.so.1 -fPIC -shared example.c -o libexample.so
(02:19:24 PM) sistpoty: the -Wl,... are commands that gcc passes to the linker
(02:20:01 PM) phoenix24: please explain a bit on them.
(02:20:22 PM) sistpoty: ok, the first one basically tells the linker, that it needs an entry for every symbol (i.e. no unresolved symbols)
(02:20:34 PM) sistpoty: note, that an "U" entry is ok here as well
(02:21:05 PM) sistpoty: because that would mean that it is linked against an shared object, which contains the symbol (the implicit libc, which gcc will always add here)
(02:21:52 PM) sistpoty: I'll come to the soname part later in the session... this option will specify that libexample.so should have the soname libexample.so.1
(02:22:10 PM) slangasek: sistpoty: might I interject a clarification regarding -Wl,-z,defs?
(02:22:15 PM) sistpoty: slangasek: sure
(02:22:47 PM) slangasek: what that option really means is that, at build time, ld must be able to find a match for each undefined symbols in one of the shared objects that you're linking against
(02:23:03 PM) slangasek: i.e, it controls *how* " U " symbols are handled
(02:23:33 PM) sistpoty: thanks for the clarification slangasek!
(02:23:41 PM) dooglus: specifically, does -Wl,z,defs mean that ld will be passed the "-z defs" flag?
(02:23:50 PM) sistpoty: dooglus: exactly
(02:23:54 PM) dooglus: thanks
(02:24:10 PM) sistpoty: -Wl,something means to pass something as an option to ld
(02:24:50 PM) sistpoty: the -fPIC will tell gcc to produce position independent code... I'll just say that it is needed to shared objects and not go into details here ;)
(02:25:06 PM) sistpoty: finally, -shared tells gcc to produce a shared object
(02:25:49 PM) sistpoty: now, to match the shared objects, you have on the system, let's strip it
(02:25:51 PM) sistpoty: strip libexample.so
(02:25:54 PM) sistpoty: +.1
(02:26:40 PM) phoenix24: +1
(02:26:42 PM) sistpoty: erm... what I wrote... (just messed things up locally *g*)
(02:26:56 PM) sistpoty: let's take a look again with nm
(02:27:01 PM) sistpoty: nm libexample.so
(02:27:17 PM) snewland: oops
(02:27:44 PM) phoenix24: No symbols : nm: libexample.so: no symbols
(02:27:59 PM) sistpoty: let's try to get it right... as it's a shared object, we're interested in the dynamic symbols
(02:28:10 PM) sistpoty: nm -D libexample.so
(02:29:00 PM) dooglus: sistpoty: can I ask a question?
(02:29:04 PM) sistpoty: dooglus: sure
(02:29:38 PM) phoenix24: +1
(02:29:57 PM) ***siretart notices that the output of eu-nm (from elfutils) is much clearer that binutil's nm
(02:30:07 PM) dooglus: sistpoty: this "so" is a shared object - has any linking been done yet?  when we were talking about options being passed to ld, has that happened yet?  the term 'shared object' makes me think it hasn't been linked yet (like a .o hasn't)
(02:30:44 PM) sistpoty: dooglus: good one
(02:30:56 PM) sistpoty: for a shared object, linking has been done
(02:31:19 PM) slangasek: yes, it's "partially" linked, with final linking happening at runtime via ld.so
(02:31:33 PM) dooglus: I thought so - I 'damaged' one of the 'fprintf's in the file, and it refused to compile any more - I was surprised.
(02:32:15 PM) siretart: there isn't much difference between an ELF shared object and an executable. the executable happens to have a function called main(), though :)
(02:32:40 PM) dooglus: ok, thanks
(02:32:41 PM) snewland: so does the ELF
(02:33:00 PM) snewland: siretart: have a main I mean
(02:33:09 PM) sistpoty: snewland: yes, because the main is still in there (s.th. libraries usually don't contain)
(02:33:20 PM) snewland: k thanks
(02:33:46 PM) slangasek: dooglus: you might like to compare the results with and without the -Wl,-z,defs
(02:34:00 PM) slangasek: (in the case of damaging the fprintf)
(02:34:36 PM) sistpoty: there are also other tools to look at the shared object's information
(02:34:47 PM) sistpoty: let's try readelf -s libexample.so
(02:35:14 PM) dooglus: with the -Wl,z,defs I see "example.c:(.text+0x56): undefined reference to `myfprintf'" - without it I see no error at all
(02:35:56 PM) sistpoty: exactly, because in the first call, the linker definitely wants to resolve myfprintf, in the second one it won't
(02:36:02 PM) snewland: dooglus: I thought the command was -Wl,-z,defs
(02:36:27 PM) dooglus: snewland: I typed it out badly by hand there.
(02:36:52 PM) sistpoty: ok, let's take a look at the output of the readelf command
(02:36:59 PM) sistpoty: anything interesting that you note?
(02:37:35 PM) snewland: size 0
(02:37:44 PM) eddyMul: @GLIBC_2.0
(02:38:30 PM) sistpoty: I'm not too sure what size 0 means exactly (probably, that it doesn't take any storage space in the shared object itself)
(02:38:38 PM) sistpoty: but let's look at what eddyMul has found
(02:38:43 PM) sistpoty: these are versioned symbols
(02:38:54 PM) sistpoty: i.e. the name of the symbol contains a version as well
(02:39:33 PM) sistpoty: oh, I should explain s.th. first
(02:39:54 PM) sistpoty: in an shared object (and a normal object as well), there can be only one symbol with a specific name
(02:40:08 PM) sistpoty: (apart from dirty linker commands getting used)
(02:40:38 PM) sistpoty: that means, you couldn't have two symbols with the name stderr
(02:40:59 PM) sistpoty: however, in the c-library uses versioned symbols, hence the version is part of the symbol
(02:41:19 PM) dooglus: GLIBC is written all in capitals; I've never seen it in capitals before
(02:41:42 PM) sistpoty: so stderr could be defined for example by an older version as well
(02:42:01 PM) sistpoty: however this symbol from our shared object would then always get resolved to the one of 2.0
(02:42:17 PM) sistpoty: (or mine against 2.2.5)
(02:42:48 PM) sistpoty: printing out symbol versions is s.th. which afaik nm doesn't do
(02:43:07 PM) sistpoty: hence, when looking at libraries, nm should be avoided
(02:43:23 PM) sistpoty: you can also use objdump to look at a shared object
(02:43:37 PM) sistpoty: it can tell other important information as well
(02:43:39 PM) sistpoty: let's try this
(02:44:03 PM) sistpoty: objdump -x libexample.so
(02:44:11 PM) sistpoty: this will produce lot's of output
(02:44:18 PM) sistpoty: objdump -p libexample.so
(02:44:30 PM) sistpoty: will give much less output with the most interesting information
(02:44:52 PM) sistpoty: so let's see what objdump -p libexample.so will tell us
(02:45:04 PM) sistpoty: the most interesting bits are
(02:45:07 PM) sistpoty: the SONAME
(02:45:29 PM) sistpoty: and the NEEDED enty (there can and usually is more than one NEEDED entry)
(02:46:08 PM) sistpoty: the SONAME entry denotes some kind of "version" of the shared object, denoting a stable abi
(02:46:35 PM) sistpoty: that means basically that you can always use a newer version of a library with the same SONAME as an older version
(02:46:48 PM) sistpoty: and your program using it will still work
(02:47:05 PM) sistpoty: anyone recalling, where the SONAME entry came from?
(02:47:41 PM) stdin: gcc -Wl,-soname...
(02:47:47 PM) sistpoty: stdin: perfect
(02:47:50 PM) AstralJava: Damn, beat me to it!
(02:48:06 PM) sistpoty: however that means, that someone set it manually
(02:48:45 PM) sistpoty: or in other words, if the person who set it did s.th. wrong, you might end up with a changed ABI
(02:49:06 PM) sistpoty: let's take a look what problems can arise by an example
(02:49:22 PM) sistpoty: first, let's get http://www.potyra.de/library_packaging/libmyhello-1.0.1.tar.gz
(02:49:48 PM) sistpoty: extract it and compile it
(02:49:56 PM) sistpoty: tar -xvzf libmyhello-1.0.1.tar.gz
(02:49:57 PM) sistpoty: make
(02:50:20 PM) sistpoty: if you've got it, please install it (with root privs):
(02:50:22 PM) sistpoty: sudo make install
(02:50:40 PM) sistpoty: no worries, it will only place stuff under /usr/local, and comes with an uninstall rule as well
(02:50:58 PM) AstralJava: We're brave people. :)
(02:51:02 PM) sistpoty: of course you can look at the makefile, if you don't trust me :P
(02:51:45 PM) sistpoty: now run ldconfig (with root privs as well), so that ld will be notified to look out for s.th. new
(02:52:18 PM) sistpoty: of course a library alone is no fun yet, so you want a program using it as well: http://www.potyra.de/library_packaging/hello_prog-1.0.tar.gz
(02:52:46 PM) sistpoty: you'll only need to compile this one
(02:53:00 PM) sistpoty: let's try it: ./hello_prog 10
(02:53:09 PM) sistpoty: amazing software, right? *g*
(02:53:33 PM) sistpoty: everyone got it so far?
(02:53:57 PM) AstralJava: Yup.
(02:54:07 PM) barcc: +1
(02:54:45 PM) sistpoty: ok, so now let's check out the new library version
(02:54:55 PM) sistpoty: http://www.potyra.de/library_packaging/libmyhello-1.0.2.tar.gz
(02:55:41 PM) sistpoty: you should know the procedure... extract, make, sudo make install
(02:56:25 PM) sistpoty: if you've got it, please try to run the application again (not to rebuild it, just run the one you've got)
(02:57:25 PM) AstralJava: Eeewwww.... symbol lookup error!
(02:57:50 PM) sistpoty: ok, let's try to find out what happened...
(02:58:01 PM) sistpoty: look at the symbols, that are undefined in hello_prog
(02:58:27 PM) sistpoty: readelf -s hello_prog
(02:58:46 PM) sistpoty: in readelf, these are listed as "UND" btw.
(02:59:01 PM) sistpoty: you'll see, that hello_prog will want a symbol called print_hello
(02:59:37 PM) sistpoty: if you look at the old shared object, there indeed is such a symbol
(02:59:59 PM) sistpoty: 7: 00000000000005a0    32 FUNC    GLOBAL DEFAULT   11 print_hello
(03:00:30 PM) sistpoty: the new library however doesn't come with one
(03:01:15 PM) sistpoty: hence, the ABI is obviously not stable
(03:01:36 PM) sistpoty: the term ABI refers to the abstract binary interface
(03:01:57 PM) sistpoty: it means, the symbols (by name) defined in the shared object and their type
(03:02:50 PM) sistpoty: for functions, it also means how the symbols can be used (i.e. how many parameters must the function have, and of which type must these be)
(03:03:35 PM) sistpoty: as you've seen, once you remove a symbol, the ABI is not stable (because any program might have used it)
(03:03:59 PM) sistpoty: the type of a symbol shouldn't change as well
(03:04:25 PM) sistpoty: this information can be easily found out with the tools you know so far
(03:04:37 PM) sistpoty: however to find out the arguments of a function call, symbol names and type alone are not sufficient
(03:04:58 PM) AstralJava: Was just about to ask. :)
(03:05:04 PM) sistpoty: one possibility is to use the debug information (which however is no longer present in binary packages)
(03:05:15 PM) sistpoty: for debug information, gdb is our tool of choice
(03:05:33 PM) sistpoty: let's try (pick one of the two library versions you like)
(03:05:49 PM) sistpoty: gdb libmyhello.so
(03:06:03 PM) sistpoty: inside gdb, you can type
(03:06:06 PM) sistpoty: info functions
(03:07:10 PM) sistpoty: you can also look at the type (as in the programming language's type, not to confuse with the symbol type) with
(03:07:12 PM) sistpoty: info variables#
(03:07:14 PM) sistpoty: -#
(03:07:39 PM) sistpoty: however this will only work, if debugging symbols are still present
(03:07:49 PM) sistpoty: you can compare this, after running strip on the shared object
(03:07:49 PM) sistpoty: you can compare this, after running strip on the shared object
(03:09:21 PM) sistpoty: ok, how about makeing a 5 minute break now, and then sum up part 1 and start with part 2?
(03:10:23 PM) AstralJava: Works for me.
(03:11:00 PM) sistpoty: ok, then let's continue at 20.15 UTC (in the hope my local clock is correct)
(03:16:04 PM) sistpoty: everyone back?
(03:17:11 PM) barcc: +1
(03:17:24 PM) sistpoty: ok, let's sum up part 1
(03:17:28 PM) AstralJava: Yep.
(03:17:43 PM) sistpoty: so far, we've learned that we can lookup the symbols via nm, readelf and objdump
(03:18:38 PM) sistpoty: a stable ABI means, that no symbols are removed and the symbols type is not changed... also that the corresponding c-convention (e.g. number of arguments of a function, type of arguments of a function, type of a variable) is not changed
(03:18:55 PM) sistpoty: in contrast (what I didn't mention yet), is the API
(03:19:21 PM) sistpoty: a stable API denotes, that a program will still be able to compile with a new library
(03:19:41 PM) sistpoty: you can have a stable API but have breakages in the ABI of course
(03:19:55 PM) sistpoty: e.g. if things get moved from a c-file to inlined version in the header file
(03:20:29 PM) elisee: (I just managed to catch up, very instructive so far :))
(03:20:31 PM) sistpoty: if the SONAME stays the same, it means that the library author *believes* that the ABI is stable
(03:20:42 PM) phoenix24_ is now known as phoenix24
(03:20:46 PM) sistpoty: but it's no guarantuee, since it's set by a human ;)
(03:21:13 PM) sistpoty: oh, one thing I missed: the NEEDED entry
(03:21:18 PM) sistpoty: or entries
(03:21:59 PM) sistpoty: the NEEDED entry in a shared object means, that a shared object with such a SONAME is needed
(03:22:18 PM) sistpoty: (likewise in a elf-binary aka program)
(03:22:42 PM) sistpoty: it's needed, because unresolved symbols, which the loader will resolve are defined there
(03:22:54 PM) sistpoty: let's take a short look at a real world example
(03:23:15 PM) sistpoty: objdump -p /usr/lib/libgnome-2.so.0
(03:23:23 PM) sistpoty: (in the hope, that everyone has this shared object)
(03:24:16 PM) sistpoty: these NEEDED entries point to shared objects (with the same SONAME), that contain symbols that libgnome-2.so.0 uses, but doesn't define
(03:24:29 PM) sistpoty: I guess you all know ldd
(03:24:32 PM) sistpoty: let's compare
(03:24:37 PM) sistpoty: ldd /usr/lib/libgnome-2.so.0
(03:25:05 PM) sistpoty: can anyone spot the difference?
(03:26:52 PM) phoenix24: sistpoty: please brief about ldd.
(03:27:09 PM) sistpoty: phoenix24: ok
(03:27:57 PM) elisee: ldd seems to return the whole dependency tree with path to the SO file
(03:28:15 PM) sistpoty: ldd will basically do the same thing, that the loader will do
(03:28:27 PM) sistpoty: i.e. it will resolve the NEEDED entries to shared objects
(03:29:04 PM) sistpoty: and as elisee wrote, will do the same for the shared objects found as dependencies as well
(03:29:22 PM) slangasek: in fact, ldd uses the loader to do what it does ;)
(03:29:35 PM) sistpoty: it will finally spit out the *pathes* to the shared object it found
(03:29:35 PM) sistpoty: heh
(03:29:58 PM) elisee: linker / load = the same program?
(03:30:07 PM) elisee: with different options
(03:30:21 PM) elisee: -load+loader
(03:30:46 PM) sistpoty: elisee: nope... the linker is used at *compile* time... to find out the NEEDED entries
(03:30:55 PM) slangasek: by "loader" here we're talking about ld.so, which is the runtime linker, yes
(03:31:27 PM) sistpoty: elisee: the loader will take the NEEDED entries as input and find the object file at *run time*
(03:31:45 PM) elisee: sistpoty, I get that
(03:31:51 PM) elisee: and ld.so is automatically loaded by the OS? because it's a shared object too, right?
(03:31:55 PM) phoenix24: got it.
(03:32:15 PM) sistpoty: elisee: that's what I assume, but I guess slangasek could tell in more detail ;)
(03:33:14 PM) slangasek: elisee: the way this works under Linux is that when you exec() a program, the kernel looks at the head of the file, sees that it's an ELF file, and passes control to ld.so to work out what to do with it
(03:33:18 PM) elisee: at least it looks like, according to the ld.so man page
(03:33:39 PM) elisee: ok thanks a lot, all of this is so instructive
(03:35:02 PM) sistpoty: finally, a very important note, which we'll dig into in part 2: shared objects with a different SONAME can be installed side-by-side
(03:35:13 PM) sistpoty: any further questions to part 1 so far?
(03:35:49 PM) elisee: nope
(03:36:03 PM) sistpoty: ok, then let's go for part 2
(03:36:28 PM) phoenix24: yes!
(03:36:36 PM) sistpoty: while my plan was to actually have everyone package libmyhello, I guess we'll never do that in time
(03:36:58 PM) sistpoty: luckily, I've got s.th. prepared, let me look at where to get it :)
(03:37:51 PM) sistpoty: bzr branch http://bazaar.launchpad.net/~sistpoty/+junk/library-packaging-session
(03:37:58 PM) sistpoty: (in the hope, that everyone has bzr installed)
(03:38:40 PM) elisee: i guess anyway it can be quickly installed by anyone through Ubuntu repositories
(03:38:46 PM) sistpoty: :)
(03:38:52 PM) sistpoty: everyone got it?
(03:39:09 PM) barcc: +1
(03:39:22 PM) elisee: ok for me
(03:39:30 PM) sistpoty: ok, since this contains the upstream code as well, we'll first need to make a tarball of it
(03:39:31 PM) phoenix24: +1
(03:39:41 PM) sistpoty: cd libmyhello/0.1 && make dist
(03:40:05 PM) ***sistpoty tries to remember the funny build system used for the package
(03:41:08 PM) elisee: I don't know much about packaging, and I wonder : does that kind of Makefile gets written by a human-being?
(03:41:23 PM) elisee: or do tools do it for us?
(03:41:28 PM) sistpoty: elisee: is this a question whether I am human? :P
(03:41:38 PM) sistpoty: (I wrote everything there by hand)=
(03:41:38 PM) LaserJock: elisee: he's not
(03:41:53 PM) elisee: sistpoty, aren't you? ;)
(03:42:02 PM) sistpoty: heh, no comment *g*
(03:42:16 PM) LaserJock: he's an advanced cyborg cron job
(03:42:53 PM) AstralJava: So AI devel is much further than They(tm) let us believe, huh?
(03:43:07 PM) phoenix24: ;)
(03:43:17 PM) elisee: :p
(03:44:21 PM) sistpoty: ok, if you've got the make dist, you'll need another dir
(03:44:34 PM) sistpoty: in libmyhello, make the directory work
(03:45:23 PM) sistpoty: if you've got that, go to the subdir packaging, and just call make (this will place some symlinks into ../work)
(03:45:50 PM) sistpoty: oh, hard links, it seems
(03:46:01 PM) sistpoty: it will also extract the packaging I prepared
(03:46:38 PM) sistpoty: everyone got it so far?
(03:47:09 PM) elisee: got it
(03:48:13 PM) sistpoty: ok, first off, before looking at anything, a *very* good resource is the debian library packaging guide... keep that under your pillow, when working on libs ;)
(03:48:33 PM) sistpoty: http://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html
(03:48:55 PM) phoenix24: subdir named "packaging ?",
(03:49:13 PM) elisee: phoenix24, library-packaging-session/libmyhello/packaging
(03:49:15 PM) sistpoty: phoenix24: libmyhello/packaging
(03:51:01 PM) sistpoty: if everyone got it, let's start with the control file library-packaging-session/libmyhello/work/libmyhello-1.0.1/debian/control
(03:52:12 PM) sistpoty: the work directory should contain the extracted package, hence libmyhello-1.0.1 in there is the directory which I'll refer from now on
(03:53:10 PM) sistpoty: in debian/control, there are two (binary) packages for this library defined
(03:53:42 PM) sistpoty: that's pretty much standard (except some libraries ship an application as well, or have big documentation)
(03:53:59 PM) sistpoty: the first one, will contain the shared object
(03:54:20 PM) sistpoty: the libmyhello1 package name is special:
(03:54:33 PM) sistpoty: it is related to the SONAME that is defined in the shared object it contains
(03:54:52 PM) sistpoty: since shared objects with a different SONAME can be installed side-by-side, the packaging must respect this
(03:55:13 PM) sistpoty: hence the relation of the package name to the SONAME, but it also has other consequences
(03:56:08 PM) sistpoty: in libmyhello1, there mustn't be a file which would have the same name (in the same directory) as a file from a shared object with a different SONAME
(03:56:29 PM) sistpoty: hence putting e.g. a manpage, or an application in there won't work
(03:57:31 PM) sistpoty: this package is the thing, which will (on the users) system usually get installed, because another package (usually an application using this shared object) draws in
(03:57:49 PM) sistpoty: others than that, there is no much use to install it directly
(03:58:32 PM) sistpoty: in contrast, the -dev package contains everything that is needed to compile programs against the shared object
(03:59:11 PM) sistpoty: this means the header files which programs must include, but also in our case the libmyhello1.so symlink
(03:59:32 PM) sistpoty: erm... sorry libmyhello.so symlink in /usr/lib
(03:59:48 PM) sistpoty: now, what's that symlink good for?
(04:00:02 PM) elisee: there's no version in it because we link against a share object and not an abi version, right?
(04:00:23 PM) sistpoty: elisee: right
(04:00:35 PM) sistpoty: usually, we'll always want to call s.th. like
(04:00:40 PM) sistpoty: gcc -lsomelib
(04:01:00 PM) sistpoty: this will make gcc to search for a shared object call libsomelib.so
(04:01:03 PM) sistpoty: in the library path
(04:01:32 PM) sistpoty: (it prepends lib and appends .so, iirc .a is also possible if no .so is found, but I'm not 100% sure on this)=
(04:02:21 PM) sistpoty: hence it makes sense to include the .so in the -dev package, so that programs will find the (right) shared object when linking
(04:02:34 PM) sistpoty: oh, for the .a: these are static libraries
(04:03:03 PM) sistpoty: since these will be part of the resulting program, and won't get loaded at run time, there is no such thing as a SONAME for these
(04:03:15 PM) sistpoty: It's simply not needed
(04:03:26 PM) elisee: so one can't name a .so file without lib prepend to it? or will it first try without the lib?
(04:03:30 PM) sistpoty: hence these are also part of the -dev package (if there)
(04:03:57 PM) elisee: without the 'lib' prefix I mean
(04:04:17 PM) sistpoty: elisee: I'm not 100% sure, but what I know: yes
(04:04:48 PM) elisee: ok
(04:04:50 PM) sistpoty: (and looking at my /usr/lib/ directory seems to underline that)
(04:05:13 PM) sistpoty: elisee: of course plugins (which are also shared objects, opened via dlopen) can use whatever names they wish
(04:05:29 PM) sistpoty: but we didn't want to look at these here ;)
(04:05:46 PM) elisee: ok
(04:06:19 PM) sistpoty: back to the control file
(04:06:34 PM) sistpoty: the -dev package must always add a (hard) dependency on the shared object
(04:06:44 PM) sistpoty: on the package containing the shared object
(04:06:53 PM) sistpoty: because you can't link anything with just a symlink ;)
(04:07:11 PM) sistpoty: that's the depends line of libmyhello-dev
(04:07:57 PM) sistpoty: others than that (and where the example is not too exact), the -dev package must also make sure, that every library, that's needed for compiling is drawn in
(04:08:47 PM) sistpoty: e.g. if I'd use stuff from libasound2 (because it's e.g. included from libmyhello headers) then there must be a dependency on libasound2-dev
(04:09:34 PM) sistpoty: oh, sometimes, it makes sense, to also name the -dev package correlating to the soname, in case you want to have more than one version of the library in the archive
(04:09:51 PM) sistpoty: e.g. if many apps won't compile with the new version
(04:12:38 PM) sistpoty: ok, so why do we need to make sure, that the library package (libmyhello1) is installable together with a package of a different SONAME (e.g. libmyhello2), even if we plan to have only one version in the archive?
(04:12:42 PM) sistpoty: anyone?
(04:13:36 PM) elisee: because the user might need two versions of the library
(04:13:44 PM) h3sp4wn_: upgrades maybe ? (either way aptitude wouldn't have that issue)
(04:13:52 PM) elisee: because he has two programs needing these two different versions
(04:14:03 PM) sistpoty: exactly elisee and h3sp4wn_
(04:14:34 PM) sistpoty: just consider, that the user has a package called hello_prog installed. It was built and got the dependency on libmyhello1
(04:14:42 PM) sistpoty: now libmyhello2 is available
(04:15:13 PM) sistpoty: the library however won't get upgraded (unless a different package needs it)
(04:15:43 PM) sistpoty: and libmyhello1 can only be uninstalled, in case hello_prog was removed
(04:16:12 PM) sistpoty: so in the archive, we'd rebuild hello_prog. That way it will pick up libmyhello2 as a dependency
(04:17:03 PM) sistpoty: then, and only then, hello_prog of the user can get upgraded (and this would draw in libmyhello2 then). However libmyhello1 wouldn't automatically go away
(04:17:13 PM) sistpoty: got it?
(04:17:35 PM) elisee: (yep)
(04:17:51 PM) AstralJava: Figured as much. So how do we get rid of deprecated libs when no app needs them anymore?
(04:18:25 PM) sistpoty: AstralJava: since these (should) always get drawn in by s.th. and not installed by hand, "apt-get autoremove" will do that trick
(04:18:27 PM) elisee: the package managers usually provide a way to remove no-longer-needed automatically installed dependencies
(04:18:58 PM) elisee: yeah, apt-get autoremove, that's what I mean
(04:19:00 PM) elisee: +t
(04:19:06 PM) AstralJava: Okay thanks!
(04:19:06 PM) elisee: :)
(04:19:42 PM) sistpoty: ok, I guess debian/rules is not too exciting. maybe apart from one call
(04:19:43 PM) sistpoty: dh_makeshlibs
(04:19:55 PM) sistpoty: this one will create a shlibs file
(04:20:40 PM) sistpoty: an shlibs file contains the info, which library package contains which shared object (together with the version of the library package that's needed for it)
(04:21:28 PM) elisee: it's built with the info we've just put in control?
(04:21:56 PM) sistpoty: this is a special part of a debian package... you can find all shlibs files of installed packages in /var/lib/dpkg/info
(04:22:28 PM) sistpoty: elisee: yes. It also can contain the info that a specific version is required (I'll come to that in a minute)
(04:22:42 PM) sistpoty: specific version of the package even
(04:23:07 PM) elisee: sorry, yet another question: can't figure out what "dh" means in dh_makeshlibs
(04:23:25 PM) sistpoty: elisee: that means it's a debhelper command
(04:23:32 PM) elisee: okay
(04:23:57 PM) sistpoty: debhelper contains commands to make common needed tasks of packaging easier
(04:24:24 PM) sistpoty: e.g. dh_shlipdebs is basically a wrapper to dpkg-shlipdeps...
(04:25:34 PM) sistpoty: which will take care that anything with ${shlibs:Depends} in debian/control will get replaced by a dependency to the library package
(04:25:39 PM) sistpoty: now how does it do that?
(04:26:24 PM) sistpoty: it looks up the NEEDED entries (with tools, we've learned in part 1) of every elf (both binaries and shared objects) and checks via the shlibs files on the system, which package it needs
(04:27:28 PM) sistpoty: now let's reconsider part 1... a stable abi (and hence a stable SONAME) means no symbols get removed and their type and meaning don't change
(04:27:54 PM) sistpoty: however upstream could add new features (e.g. new functions) which would result in new symbols
(04:28:01 PM) sistpoty: and the ABI would still stay the same
(04:28:27 PM) sistpoty: i.e. programs that use the old shared object can use the new shared object
(04:28:47 PM) sistpoty: but what, if a program uses exactly one of the new symbols?
(04:29:38 PM) sistpoty: it could then not run with an old shared object not containing the symbol yet, which however is not different in regards to the soname
(04:29:56 PM) sistpoty: hence on the mere basis of an SONAME there doesn't seem to be a solution
(04:30:05 PM) sistpoty: luckily the debian packaging system can help here
(04:30:22 PM) sistpoty: as I stated earlier, the shilbs file may also contain a version
(04:30:52 PM) sistpoty: you can specify the version that goes in there with the -V parameter of dh_makeshlibs
(04:31:17 PM) sistpoty: so, in case there are new symbols in a package, you'll always want to use that:
(04:32:07 PM) sistpoty: any program, that uses the newer functionality, cannot compile against the old version (we've seen what happens with -Wl,-z,defs)
(04:32:37 PM) sistpoty: so it will need to compile the newer version of the -dev package... and the shlibs mechanism contains then a shlibs file which will state:
(04:32:51 PM) sistpoty: you'll need a this version or a later one of lib...
(04:33:43 PM) sistpoty: was that too fast?
(04:33:51 PM) elisee: so there's a numbering scheme inside the shlibs different from the SONAME version?
(04:34:18 PM) slangasek: correct
(04:34:33 PM) slangasek: SONAME documents each backwards-incompatible ABI change
(04:34:48 PM) slangasek: shlibs gives you the other piece, to document backwards-compatible ABI changes
(04:34:48 PM) elisee: I don't really know what these shlibs are in fact, but maybe it's not the purpose to explain this now
(04:35:10 PM) sistpoty: oh, then I *was* too fast
(04:35:23 PM) sistpoty: let's try again...
(04:35:56 PM) sistpoty: if you've got a shared library with a given soname, the shlibs file will basically say in what debian package this can be found
(04:36:25 PM) sistpoty: but it will also say what version of the debian package you need (at least)
(04:37:05 PM) sistpoty: in case a program gets built, which contains NEEDED entries (i.e. uses symbols from some shared object)
(04:37:38 PM) sistpoty: the shlibs file will be used to lookup in what package the shared object is
(04:37:49 PM) elisee: ok, that's much clearer this way for me
(04:38:23 PM) sistpoty: this is then added as a dependency to the programs package (actually ${shilbs:Depends} in debian/control will get replaced by the found entries)
(04:38:33 PM) sistpoty: ok, any other questions?
(04:39:05 PM) AstralJava: Probably loads, but they might pop up in practice. Can't think of any right now.
(04:39:45 PM) sistpoty: ok, then I guess one final hint (as I've once again used more time than available):
(04:40:20 PM) sistpoty: as we learned today, the most difficult bit is *upgrading* a library package
(04:40:40 PM) sistpoty: there, you must ensure, that the ABI doesn't break
(04:41:12 PM) sistpoty: of course you should also keep a look what packages build against this package (not that you'll upgrade it and every using package won't build any longer)
(04:41:24 PM) sistpoty: but the tricky part is to ensure ABI stability
(04:41:41 PM) sistpoty: I hope, you learned some tools with which you can check that today
(04:42:10 PM) AstralJava: Definitely! What a fantastic session it was.
(04:42:14 PM) elisee: thanks a lot :)
(04:42:32 PM) sistpoty: others than that, there exists a package which can be a help, but I forgot it's name... slangasek: what creates the manifest again?
(04:45:40 PM) sistpoty: ok, sorry I really have completely forgotten the name of the package...
(04:46:01 PM) sistpoty: however thanks for coming, and I hope once you'll maintain a library package, you'll be prepared ;)
(04:46:07 PM) AstralJava: Can you post it on ubuntu-motu@ or something later on?
(04:46:36 PM) sistpoty: AstralJava: you mean which package I mean?
(04:46:38 PM) AstralJava: Again, thank you very much for this, hugely appreciated! :)
(04:47:11 PM) AstralJava: sistpoty: That's right, if/when you recall or somebody else verifies it.
(04:47:30 PM) sistpoty: it must be in the old logs... but I'll post it ;)
(04:47:56 PM) slangasek: sistpoty: manifests are created by dpkg-dev itself now, no?  dpkg-gensymbols?
(04:48:22 PM) slangasek: or icheck, for API manifests
(04:48:24 PM) sistpoty: slangasek: no, that package which parses the c-code and produces a .manifest file
(04:48:31 PM) slangasek: icheck
(04:48:38 PM) sistpoty: thanks!
(04:48:46 PM) AstralJava: Gotcha.
(04:49:10 PM) AstralJava: Thanks, Stefan and Steve.
(04:49:17 PM) sistpoty: (strange enough I was searching for iwcheck the whole time... close but no hit)
(04:49:30 PM) sistpoty: thanks for listening ;)
(04:49:56 PM) sistpoty: now, I'll have a cool drink :)
(04:50:26 PM) AstralJava: Well deserved it too. :)