== Dev Week -- Java library packaging with maven-debian-helper -- jamespage -- Thu, Jul 14th, 2011 == {{{#!irc [20:01] Hi Everyone o/ [20:02] Welcome to the Java library packaging with maven-debian-helper Ubuntu Developer Week session [20:02] So - organisational bits first: [20:02] Please ask questions whenever you like on #ubuntu-classroom-chat and please prefix your question with 'QUESTION:' I'll try to keep and eye out [20:02] All session logs will be uploaded and accessible from https://wiki.ubuntu.com/UbuntuDeveloperWeek - so if you need a reminder of something covered in the session then please look there. [20:03] My name is James Page and I've been contributing to Ubuntu since September last year [20:03] I work for Canonical and I'm a member of the Ubuntu Server team so apologies if some of this session is a little server centric :-) [20:04] This session is intended to give you an overview of packaging Java libraries for Ubuntu - specifically Java projects that use the Maven build system. [20:04] So first things first - why package Java libraries for Ubuntu? [20:05] Daviey asked: Java tradionally has a bad reputation on Linux, why is this? [20:05] Timely question - lets cover that one first [20:05] I think it due to the way the Java projects are typically developed [20:06] Because the Java Virtual Machine abstracts the developer away from the underlying operating system [20:06] they are free to load that virtual machine with exactly the code they want [20:07] dependent libraries are often bundled in a variety of nasty ways [20:07] which means that its hard to pull out common libraries and base line in the way a Linux distro likes todo. [20:08] not impossible - just hard [20:08] to a certain extent the toolsets exacerbate this behaviour [20:09] so back to my original question - why package Java libraries for Ubuntu? [20:09] Well from my perspective its about delivering Java applications onto the Ubuntu platform; [20:09] this would include things like Tomcat, Jenkins, Hadoop, Eclipse etc... [20:09] (some of which we have in Ubuntu, some of which we don't - yet) [20:10] However these applications need to rely on the Java libraries that are packaged for Ubuntu [20:10] in the same way that other libraries are packaged once and then used by all applications that need them we aim todo the same with Java libraries. [20:11] So having a broad, well maintained set of libraries is key to both maintaining the existing Java applications and delivering new applications into Ubuntu. [20:11] This does not always align well with the way that most Java project work; [20:12] as I just discussed each project can pretty much load the Java Virtual Machine with the code then want [20:12] which means they don't have to give consideration to other Java projects - because they will be isolated in the same way [20:13] however by maintaining a single set of libraries we try to bring some of the Linux distro goodness to the Java world [20:13] So - next bit is a little dull but important [20:13] Java libraries should follow the Debian Java Policy - see http://www.debian.org/doc/packaging-manuals/java-policy/ for the full details. [20:14] In fact its probably worth pointing out now that 90%+ of the Java libraries come unchanged for Debian [20:14] As a quick 101: [20:14] Library packages should be named libXXX-java - for example libcommons-io-java [20:14] Documentation should be in a separate package libXXX-java-doc [20:14] Libraries (jars in the case of Java) should be installed into /usr/share/java - publishing Jar's into a single location aids with discover-ability etc... [20:14] and where possible into /usr/share/maven-repo - more on this in a bit. [20:15] Fortunately the toolset around generating Java library packaging for Maven projects helps out quite alot with this so you should find it fairly easy to generate a policy compliant library. [20:16] Just in case you are not familiar with Maven its probably the most popular software project management tool used by Java projects [20:16] Its much more than just a build tool - hence 'Software Project Management' [20:17] through a pretty extensive range of plugins it integrates with most SCM's and issue trackers allowing easy automation of defect tracking, release processes, publishing of project artefacts etc... [20:18] however in Ubuntu it does get pretty much relegated to being a build tool - most of value add is not applicable and is mainly used by the upstream projects [20:18] It operates by convention; if you stick your code in the right place Maven will find it and build it; it will also have a look for test code and compile and run that as well. [20:19] Its uses project object model files - POM's [20:19] to define various metadata about the software project including its namespave, name, dependencies, build process, author, licensing .... [20:20] the list goes on and is incredibly rich - this information is really useful when packaging [20:20] but depth is not always great :-( [20:20] So I'm now going to give a quick demo of packaging a basic Java library using maven-debian-helper [20:20] To follow the demo you will need to log into the following server [20:20] ssh guest@ec2-46-137-134-25.eu-west-1.compute.amazonaws.com [20:20] password should be guest [20:21] I'm going to be using a tool called byobu so that we can all see the same session. [20:21] So as it takes a bit of time I've already setup the packaging environment on this Ubuntu server; [20:22] If you needed to do it [20:22] sudo apt-get install packaging-dev default-jdk maven-debian-helper javahelper apt-file aptitude [20:22] packaging-dev (thanks bdrung) is a new package which is intended to setup all the basics for package development; [20:22] we also need the Java Dev Kit and the java specific helpers (plus some other tools) [20:22] So next step - generate some basic packaging using mh_make; [20:23] we are going to pull a upstream tarball from github and then do some basic packaging [20:24] metainf-services is a very simple library that helps generated metadata for jar files during packaging [20:24] so as you can see we just have a pom.xml file and a src dir [20:25] next we use mh_make [20:25] mh_make tries to guess most things but it does give you the option to change stuff [20:26] so we are going to run tests and generate API docs [20:26] mh_make uses apt-file to search for any missing dependencies [20:26] if it can't find something it needs in /usr/share/maven-repo it will search using apt-file and try to make a recommendation [20:27] So this bit is quite important [20:27] when the package is built it gets deployed into /usr/share/maven-repo twice [20:27] once under the original version - 1.2 in this case [20:28] and once under a fixed label - this is normally debian or 1.x/2.x if multiple versions of a library are packaged [20:28] this means that other libraries can 'fix' on a version which can then be changed under the hood if a new version of the library is released [20:28] without having to update all pom files [20:28] more on this in a bit [20:29] mh_make also makes a guess as to with plugins are not useful for packaging [20:29] this last one is unknow - infact its used for publishing a micro-site to github - which we don't need either [20:30] mh_make then generates the base packaging [20:30] you will notice that it used licensecheck to search for useful information on copyright and licensing - we'll see the results of that in a mo [20:31] So I'm going to make this into a bazaar branch to help out a bit [20:31] So lets take a look around: [20:32] So using the information that licensecheck found in the headers mh_make has had a stab at generating a copyright file [20:32] its not bad - normally this needs a few tweaks to get it exactly right but it does most of the hard work [20:32] The maven.*rules files are used by maven-debian-helper to transform the Maven pom.xml files during the build of the project; [20:33] so in maven.ignoreRules you can see the two plugins that we told mh_make to ignore - these get transformed out of the pom.xml during the build [20:34] maven.rules is pretty simple in this case - this will create a 1.x version alongside the 1.2 version in /usr/share/maven-repo [20:34] so we also get a maven.cleanIgnoreRules [20:34] this is used when the clean target is called for the project [20:35] typically this requires alot less dependencies so it may be a longer list of exclusions [20:35] for this package its OK for it to be the same as maven.ignoreRules [20:35] so I'll just do a quick change to the changelog and we are good to go [20:36] You will notice that mh_make has used Java Demo [20:36] this gets picked up from DEBEMAIL and DEBFULLNAME environment variabled - I set these up earlier [20:37] so source package build first [20:38] maven-debian-helper patches/unpatches the pom.xml file during the process [20:38] the default target for building the package is 'package' - however for more complex packages 'install' may be more appropriate. [20:38] generating javadoc [20:38] (twice - this is a bug!) [20:39] automatically determining dependencies for the binary packages [20:40] and done - a few lintian warnings but nothing unsolvable [20:40] so we now have a built package \o/ [20:41] as we discussed earlier the package deploys 1.2 and 1.x artefacts to /usr/share/maven-repo [20:42] it also deploys to /usr/share/java - this is to support ant/javahelper based build and applications that require this library [20:42] and also a -doc package containing the generated API [20:42] just to prove that we have created something that at least installs [20:43] Obviously this is a relatively simple example; [20:43] however the concepts covered in this session can be used to build up complex dependency chains of packages to support large Java applications like Jenkins. [20:43] Maven actually makes packaging for Ubuntu easier; [20:44] because it has a standard way of expressing dependencies and a rich metadata model it is easier to automate the production of packaging [20:44] packages that use ant are more challenging as they express dependencies in a million bespoke ways [20:44] (unless they are using a dependency management system like ivy which can make things a little easier). [20:45] Questions? [20:46] coalitians asked: so mavens central repository is never used ? [20:46] good question - the local Debian repo in /usr/share/maven-repo is used and Maven is executed fully offline [20:47] maven-debian-helper handles this as part of the packaging build rules [20:47] so if you switch back to the demo you can see that the debian/rules file is very simple - all of the logic in encapsulated in cdbs classes [20:48] Any more for any more? [20:48] OK [20:48] I hope that you have found this session useful; [20:49] I normally hang out on #ubuntu-server and #ubuntu-java so if you have any more questions please feel free to ping me. [20:49] Thankyou and goodnight! }}}