Ubuntu Wiki

   1 [12:59] <ttx> Who is here for the Packaging Training session on Java libraries ?
   2 [13:00] <ttx> anyone? :)
   3 [13:01] <jkeyes0> I am, sort of. I know little to nothing about packaging in Ubuntu, so I just planned to sit on the sidelines and observe.
   4 [13:01]  * drubin 
   5 [13:01] <ttx> jkeyes0: that's perfectly alright. I was worried I would talk with noone listening
   6 [13:02] <ttx> OK, let's start !
   7 [13:02] <ttx> Welcome everyone to this packaging training session, today the subject is Java library packaging...
   8 [13:02] <ttx> This session assumes you know some basics of Debian packaging and have a vague notion of what a Java library and "ant" are.
   9 [13:02]  * Quintasan waves
  10 [13:02] <ttx> Talk / discussion should happen in #ubuntu-classroom-chat. Prefix questions with QUESTION: so that I can spot them more easily. Let me know on channel if I go too fast...
  11 [13:03] <ttx> So...
  12 [13:03] <ttx> Java software uses lots of Java libraries, which are commonly provided as JAR files
  13 [13:03] <ttx> It usually ships with a directory full of those JAR files and some launcher that builds a classpath from them. But that's not how we do it in Linux distributions
  14 [13:03] <ttx> We want to build from source, and we want to avoid code duplication. So we want to package Java libraries separately, place them in a common directory (/usr/share/java), and then have the software started with a classpath that points to them
  15 [13:04] <ttx> Trying to have a common library set causes several global issues (in particular with software requiring slightly different versions of the same library) but that's not the purpose of this session
  16 [13:04] <ttx> We lack a lot of very common Java libraries, which makes packaging Java software more complex than it should. This session will focus on the specific rules and tips to properly package Java libraries, with a few examples.
  17 [13:05] <ttx> Any question so far ?
  18 [13:05] <ttx> OK, there should be a lot more in a few minutes :)
  19 [13:05] <ttx> So let's say you are packaging some Java software and it imports org.safehaus.uuid, conveniently from a binary JAR bundled with its own source in lib/jug-asl-2.2.0.jar
  20 [13:06] <ttx> I meant, "So let's say you are packaging some Java software and it imports org.safehaus.uuid, conveniently from a binary JAR bundled with its own source in lib/jug-asl-2.2.0.jar"
  21 [13:06] <ttx> We can't just use that binary JAR. We have to build that JAR from source and have the software build-depend on it.
  22 [13:07] <ttx> So we need to package version 2.2.0 of that "jug-asl" library.
  23 [13:07] <ttx> Basically what you want to do is to build jug-asl.jar from source and end up with /usr/share/java/jug-asl-2.2.0.jar, together with a symbolic link /usr/share/java/jug-asl.jar -> jug-asl-2.2.0.jar
  24 [13:08] <ttx> First you need to double-check that this library hasn't already been packaged somewhere. It's not as easy as it sounds, sometimes its classes will be hidden somewhere in a strange jar name in an obscure package...
  25 [13:08] <ttx> To be sure, you would need to unjar all the JARs that the distribution contains to check.
  26 [13:08] <ttx> Fortunately I did that for you and published java-Contents.gz files that contain all classes in all jars in all packages...
  27 [13:08] <ttx> See https://wiki.ubuntu.com/JavaTeam/JavaContents
  28 [13:09] <ttx> So making sure that nobody already provides org.safehaus.uuid in Jaunty is just a matter of downloading the java-Contents.gz file and do a "zgrep org.safehaus.uuid jaunty-java-Contents.gz"
  29 [13:10] <ttx> If you try it, you can see that it's packaged as part of the "eucalyptus-javadeps" package...
  30 [13:10] <ttx> But now you want to properly package it separately, depending on eucalyptus-javadeps is just not an option for you.
  31 [13:10] <ttx> Questions, so far ?
  32 [13:11] <ttx> Before we attack the packaging part :)
  33 [13:12] <ttx> I'll assume everything is crystal clear, then.
  34 [13:12] <ttx> QUESTION: how could I know that an app uses org.safehaus.uuid?
  35 [13:13] <ttx> You can usually tell from the build documentation of the software you want to package.
  36 [13:13] <ttx> But you often find out when you try to compile it... and get import errors during java compilation
  37 [13:14] <ttx> jmehdi: does that answer your question ?
  38 [13:14] <jmehdi> ttx: yep ;)
  39 [13:15] <ttx> Ok, let's continue.
  40 [13:15] <ttx> Java library packaging follows the usual rules, with a few specificities.
  41 [13:15] <ttx> First debian/control:
  42 [13:15] <ttx> You should build-depend on "default-jdk" (+any other Java library needed at build-time)
  43 [13:15] <ttx> That will use openjdk-6-jdk in Ubuntu
  44 [13:16] <ttx> (for amd64/i386)
  45 [13:16] <ttx> The binary package that will install those MUST be named libjug-asl-java, this is a requirement of the Debian Java policy. I'll name the source package after the project name, "jug", even if some packagers use libfoo-java as the source package name as well.
  46 [13:17] <ttx> The binary should usually depend on "default-jre-headless | java2-runtime-headless". That makes sure the right runtime environment is pulled if nothing is already present.
  47 [13:17] <ttx> If the library requires Java 5 or Java 6, use java5- or java6- instead of java2-
  48 [13:17] <ttx> If the library *really* cannot do anything without the full JRE, then you should depend on "default-jre | java2-runtime". But usually you don't need to.
  49 [13:18] <ttx> Any question about the debian/control file ?
  50 [13:19] <ttx> QUESTION: what's the difference between jre-headless and default-jre?
  51 [13:20] <ttx> The "headless" JRE is one without graphical extensions
  52 [13:20] <ttx> so it doesn't pull 50 Mb of X libraries with it
  53 [13:20] <ttx> On a desktop you don't really care
  54 [13:21] <ttx> but on servers... you clearly don't want to install all those X libraries that your Java library doesn't really use
  55 [13:21] <ttx> OK, moving on to debian/rules
  56 [13:21] <ttx> the complex part :)
  57 [13:21] <ttx> Downloading jug-src.tar.gz from http://jug.safehaus.org/Download I can see that there is a build.xml there, which means it uses the ant build system, which is good news.
  58 [13:22] <ttx> "ant" is like a Java-oriented "make". The Makefile is called build.xml and contains targets (in XML) that you can call with ant <target>.
  59 [13:22] <ttx> The easy way to write the rules file is to use the CDBS ant class. See http://pastebin.ubuntu.com/192289/ for example
  60 [13:22] <ttx> simple, uh ?
  61 [13:23] <ttx> You just need to identify the ant task that will build what you need. here it is "jar.asl", you don't want the native part of the jug-lgpl version.
  62 [13:23] <ttx> Now, the tricky part.
  63 [13:23] <ttx> To successfully compile you will need to have the required Java libraries in the compiler classpath.
  64 [13:24] <ttx> To make the Ubuntu build-daemons happy we need to build-depend on the needed library package and insert its JAR into the compiler classpath, we can't use directly another binary coming from nowhere.
  65 [13:24] <ttx> As a sidenote, that means you first need to properly package all the Java library dependencies needed to build your own... which can be a very large number ;)
  66 [13:24] <ttx> The CDBS ant class has a nifty mechanism for inserting JARs in the system classpath, just add the required JAR names on a DEB_JARS line. You can put JAR names (without path or extension, it will look in /usr/share/java) or complete paths.
  67 [13:25] <ttx> In our example, jug needs the log4j library to compile. We would just add a liblog4j-java build-depend in debian/control, and in debian/rules :
  68 [13:25] <ttx> DEB_JARS := log4j
  69 [13:26] <ttx> (will add /usr/share/java/log4j.jar to the Java compiler classpath)
  70 [13:26] <ttx> If the build.xml doesn't try to be too smart with required dependencies, it's all that is needed. But sometimes the "source" bundles the needed binary JAR dependencies, so you need to patch out the part that adds them to the classpath.
  71 [13:26] <ttx> Another option is to play with the "build.sysclasspath" ant property to control precedence between system classpath and build.xml classpath, like this:
  72 [13:27] <ttx> DEB_ANT_ARGS := -Dbuild.sysclasspath=last
  73 [13:27] <ttx> There is no definitive solution since it all depends on how the build.xml file is written. I'd suggest starting packaging easy libraries with no build-dependencies, then try one with a couple dependencies, etc.
  74 [13:27] <ttx> Questions ?
  75 [13:28] <ttx> That may seem complex. But Libraries without any dependencies are really simple
  76 [13:29] <bdrung> i missed the start, is there already a log for this session?
  77 [13:29] <ttx> QUESTION: is this dependencies management specific to java?
  78 [13:29] <ttx> bdrung: maybe on irclogs.ubuntu.com
  79 [13:30] <ttx> This dependency management is not really specific to Java
  80 [13:30] <ttx> What is specific to Java is that stuff usually depends on tons of other stuff
  81 [13:31] <ttx> and their build systems aren't distribution-oriented
  82 [13:31] <ttx> so they just pack their binary dependencies with their source code
  83 [13:31] <ttx> and tell you to run "ant" and be done with it.
  84 [13:31] <ttx> the difficulty here is to make that build-daemon compatible
  85 [13:32] <ttx> and end up with everything built from pure source
  86 [13:32] <ttx> and without code duplication
  87 [13:32] <ttx> That's where it gets complex :)
  88 [13:33] <ttx> OK, we have some time left, so I'll speak of what we do when the library doesn't use "ant".
  89 [13:33] <ttx> If there is no build system, the easiest way is to write a minimal build.xml file and use the ant CDBS class.
  90 [13:33] <ttx> Something like http://pastebin.ubuntu.com/192303/
  91 [13:34] <ttx> (this one will compile all classes in src/main/java and make a JAR out of them)
  92 [13:34] <ttx> This should be sufficient for most libraries. But sometimes... if you're unlucky... the stuff you want to package will use maven.
  93 [13:34]  * ttx pauses for some dramatic effect
  94 [13:35] <ttx> Maven is a complex build/integration system that handles both build rules and dependency management. The trouble for us is that it makes heavy use of pre-built JARS that it downloads directly from Maven repositories.
  95 [13:35] <ttx> Since we want reproduceability our build-daemons cannot download from outside repositories. We need to use our own packages. So we need to bypass Maven dependency management completely.
  96 [13:35] <ttx> If the software is simple enough I'd recommend rewrite an ant build.xml file, as if there were no build system, and use the maven files as hints on required dependencies.
  97 [13:36] <ttx> But if you're really really unlucky, the thing you're trying to package will make heavy use of "maven plugins" so rewriting it as a build.xml file is just impossible.
  98 [13:36]  * ttx pauses again for more dramatic effect
  99 [13:36] <ttx> We are still investigating techniques to work around that problem. The idea is to set up an environment around maven to make it believe it already has everything it needs to run, so that it doesn't download anything.
 100 [13:37] <ttx> For those interested, Debian proposes: http://wiki.debian.org/Java/MavenBuilder. It's not really complete, but they packaged a lot of Maven plugins already so that's very good.
 101 [13:37] <ttx> We originally proposed a slightly different approach, less integrated but that can be used on a per-package basis: https://wiki.ubuntu.com/JavaTeam/Specs/MavenSupportSpec. We benefit from Debian work on packaging plugins.
 102 [13:38] <ttx> That's all about alternative build systems.
 103 [13:38] <ttx> To sum it up, looking at the build system and dependencies you'll have a good idea of the complexity of the task ahead
 104 [13:39] <ttx> ant-based, no build dependencies --> easy
 105 [13:39] <ttx> ant-based, a few already-packaged build dependencies --> medium
 106 [13:39] <ttx> no build system --> medium
 107 [13:40] <ttx> ant-based or no build system, lots of build deps --> hard
 108 [13:40] <ttx> maven-based with lots of plugins --> huh
 109 [13:40] <ttx> Questions ?
 110 [13:41] <ttx> QUESTION: does eclipse uses maven?
 111 [13:42] <ttx> I don't think so. I think they use ant, and they don't have so much external dependencies
 112 [13:42] <ttx> apt-get source eclipse and looking at debian/rules you should be able to see
 113 [13:43] <ttx> Jetty 6 uses maven, for example
 114 [13:43] <ttx> But upstream was kind enough to produce an ant build.xml
 115 [13:44] <ttx> Most of the stuff I talked about is documented in wiki at https://wiki.ubuntu.com/JavaTeam/LibraryPackaging
 116 [13:44] <ttx> Please join #ubuntu-java and the Java team meetings (on Thursdays, 1400 UTC) if you're interested in doing Java packaging.
 117 [13:45] <ttx> We should publish a list of wanted libraries, with an evaluation of the expected difficulty
 118 [13:45] <darkwise> Q: is-it the same why to package a java program ? (there is some where to ask questions?)
 119 [13:45] <darkwise> ah ok thanks :)
 120 [13:46] <ttx> darkwise: Java programs have the same problems. A few things I talked about don't apply to them though
 121 [13:46] <ttx> like package naming
 122 [13:46] <ttx> or runtime dependencies
 123 [13:46] <ttx> But in the end programs are made of libraries, so... :)
 124 [13:47] <ttx> If you look into the tomcat6 package for example...
 125 [13:47] <ttx> it produces a library part called libtomcat6-java
 126 [13:47] <ttx> the rest is the launchers, init scripts and tools around that
 127 [13:48] <ttx> that's all I wanted to talk about, so I open the floor for discussion
 128 [13:48] <darkwise> I get it ;) thanks ttx
 129 [13:49] <ttx> Any other question ?
 130 [13:50] <ttx> OK, thanks everyone for listening, hopefully I didn't make it look like more difficult than it really is :)
 131 [13:51] <ttx> I expect the logs of the session to be available at http://irclogs.ubuntu.com/2009/06/11/%23ubuntu-classroom.txt in a few minutes
 132 [13:51] <darkwise> thank you ttx
 133 [13:51] <ttx> for those that have been missing the beginning
 134 [13:51] <maxpaguru> ttx: thanks! Bye everybody
 135 [13:52] <ttx> and soon on https://wiki.ubuntu.com/Packaging/Training/Logs
 136 [13:52] <derwas> nice!
 137 [13:53] <ttx> For more questions you can always ping me on #ubuntu-java
 138 [13:53] <ttx> (as long as it's Java packaging questions more than Java questions)
 139 [13:54] <ttx> bye everyone !