Dev Week -- Java library packaging with maven-debian-helper -- jamespage -- Thu, Jul 14th, 2011
1 [20:01] <jamespage> Hi Everyone o/ 2 [20:02] <jamespage> Welcome to the Java library packaging with maven-debian-helper Ubuntu Developer Week session 3 [20:02] <jamespage> So - organisational bits first: 4 [20:02] <jamespage> Please ask questions whenever you like on #ubuntu-classroom-chat and please prefix your question with 'QUESTION:' I'll try to keep and eye out 5 [20:02] <jamespage> All session logs will be uploaded and accessible from https://wiki.ubuntu.com/UbuntuDeveloperWeek - so if you need a reminder of something covered in the session then please look there. 6 [20:03] <jamespage> My name is James Page and I've been contributing to Ubuntu since September last year 7 [20:03] <jamespage> I work for Canonical and I'm a member of the Ubuntu Server team so apologies if some of this session is a little server centric :-) 8 [20:04] <jamespage> This session is intended to give you an overview of packaging Java libraries for Ubuntu - specifically Java projects that use the Maven build system. 9 [20:04] <jamespage> So first things first - why package Java libraries for Ubuntu? 10 [20:05] <ClassBot> Daviey asked: Java tradionally has a bad reputation on Linux, why is this? 11 [20:05] <jamespage> Timely question - lets cover that one first 12 [20:05] <jamespage> I think it due to the way the Java projects are typically developed 13 [20:06] <jamespage> Because the Java Virtual Machine abstracts the developer away from the underlying operating system 14 [20:06] <jamespage> they are free to load that virtual machine with exactly the code they want 15 [20:07] <jamespage> dependent libraries are often bundled in a variety of nasty ways 16 [20:07] <jamespage> which means that its hard to pull out common libraries and base line in the way a Linux distro likes todo. 17 [20:08] <jamespage> not impossible - just hard 18 [20:08] <jamespage> to a certain extent the toolsets exacerbate this behaviour 19 [20:09] <jamespage> so back to my original question - why package Java libraries for Ubuntu? 20 [20:09] <jamespage> Well from my perspective its about delivering Java applications onto the Ubuntu platform; 21 [20:09] <jamespage> this would include things like Tomcat, Jenkins, Hadoop, Eclipse etc... 22 [20:09] <jamespage> (some of which we have in Ubuntu, some of which we don't - yet) 23 [20:10] <jamespage> However these applications need to rely on the Java libraries that are packaged for Ubuntu 24 [20:10] <jamespage> in the same way that other libraries are packaged once and then used by all applications that need them we aim todo the same with Java libraries. 25 [20:11] <jamespage> So having a broad, well maintained set of libraries is key to both maintaining the existing Java applications and delivering new applications into Ubuntu. 26 [20:11] <jamespage> This does not always align well with the way that most Java project work; 27 [20:12] <jamespage> as I just discussed each project can pretty much load the Java Virtual Machine with the code then want 28 [20:12] <jamespage> which means they don't have to give consideration to other Java projects - because they will be isolated in the same way 29 [20:13] <jamespage> however by maintaining a single set of libraries we try to bring some of the Linux distro goodness to the Java world 30 [20:13] <jamespage> So - next bit is a little dull but important 31 [20:13] <jamespage> Java libraries should follow the Debian Java Policy - see http://www.debian.org/doc/packaging-manuals/java-policy/ for the full details. 32 [20:14] <jamespage> In fact its probably worth pointing out now that 90%+ of the Java libraries come unchanged for Debian 33 [20:14] <jamespage> As a quick 101: 34 [20:14] <jamespage> Library packages should be named libXXX-java - for example libcommons-io-java 35 [20:14] <jamespage> Documentation should be in a separate package libXXX-java-doc 36 [20:14] <jamespage> Libraries (jars in the case of Java) should be installed into /usr/share/java - publishing Jar's into a single location aids with discover-ability etc... 37 [20:14] <jamespage> and where possible into /usr/share/maven-repo - more on this in a bit. 38 [20:15] <jamespage> Fortunately the toolset around generating Java library packaging for Maven projects helps out quite alot with this so you should find it fairly easy to generate a policy compliant library. 39 [20:16] <jamespage> Just in case you are not familiar with Maven its probably the most popular software project management tool used by Java projects 40 [20:16] <jamespage> Its much more than just a build tool - hence 'Software Project Management' 41 [20:17] <jamespage> through a pretty extensive range of plugins it integrates with most SCM's and issue trackers allowing easy automation of defect tracking, release processes, publishing of project artefacts etc... 42 [20:18] <jamespage> however in Ubuntu it does get pretty much relegated to being a build tool - most of value add is not applicable and is mainly used by the upstream projects 43 [20:18] <jamespage> It operates by convention; if you stick your code in the right place Maven will find it and build it; it will also have a look for test code and compile and run that as well. 44 [20:19] <jamespage> Its uses project object model files - POM's 45 [20:19] <jamespage> to define various metadata about the software project including its namespave, name, dependencies, build process, author, licensing .... 46 [20:20] <jamespage> the list goes on and is incredibly rich - this information is really useful when packaging 47 [20:20] <jamespage> but depth is not always great :-( 48 [20:20] <jamespage> So I'm now going to give a quick demo of packaging a basic Java library using maven-debian-helper 49 [20:20] <jamespage> To follow the demo you will need to log into the following server 50 [20:20] <jamespage> ssh firstname.lastname@example.org 51 [20:20] <jamespage> password should be guest 52 [20:21] <jamespage> I'm going to be using a tool called byobu so that we can all see the same session. 53 [20:21] <jamespage> So as it takes a bit of time I've already setup the packaging environment on this Ubuntu server; 54 [20:22] <jamespage> If you needed to do it 55 [20:22] <jamespage> sudo apt-get install packaging-dev default-jdk maven-debian-helper javahelper apt-file aptitude 56 [20:22] <jamespage> packaging-dev (thanks bdrung) is a new package which is intended to setup all the basics for package development; 57 [20:22] <jamespage> we also need the Java Dev Kit and the java specific helpers (plus some other tools) 58 [20:22] <jamespage> So next step - generate some basic packaging using mh_make; 59 [20:23] <jamespage> we are going to pull a upstream tarball from github and then do some basic packaging 60 [20:24] <jamespage> metainf-services is a very simple library that helps generated metadata for jar files during packaging 61 [20:24] <jamespage> so as you can see we just have a pom.xml file and a src dir 62 [20:25] <jamespage> next we use mh_make 63 [20:25] <jamespage> mh_make tries to guess most things but it does give you the option to change stuff 64 [20:26] <jamespage> so we are going to run tests and generate API docs 65 [20:26] <jamespage> mh_make uses apt-file to search for any missing dependencies 66 [20:26] <jamespage> if it can't find something it needs in /usr/share/maven-repo it will search using apt-file and try to make a recommendation 67 [20:27] <jamespage> So this bit is quite important 68 [20:27] <jamespage> when the package is built it gets deployed into /usr/share/maven-repo twice 69 [20:27] <jamespage> once under the original version - 1.2 in this case 70 [20:28] <jamespage> and once under a fixed label - this is normally debian or 1.x/2.x if multiple versions of a library are packaged 71 [20:28] <jamespage> this means that other libraries can 'fix' on a version which can then be changed under the hood if a new version of the library is released 72 [20:28] <jamespage> without having to update all pom files 73 [20:28] <jamespage> more on this in a bit 74 [20:29] <jamespage> mh_make also makes a guess as to with plugins are not useful for packaging 75 [20:29] <jamespage> this last one is unknow - infact its used for publishing a micro-site to github - which we don't need either 76 [20:30] <jamespage> mh_make then generates the base packaging 77 [20:30] <jamespage> you will notice that it used licensecheck to search for useful information on copyright and licensing - we'll see the results of that in a mo 78 [20:31] <jamespage> So I'm going to make this into a bazaar branch to help out a bit 79 [20:31] <jamespage> So lets take a look around: 80 [20:32] <jamespage> So using the information that licensecheck found in the headers mh_make has had a stab at generating a copyright file 81 [20:32] <jamespage> its not bad - normally this needs a few tweaks to get it exactly right but it does most of the hard work 82 [20:32] <jamespage> The maven.*rules files are used by maven-debian-helper to transform the Maven pom.xml files during the build of the project; 83 [20:33] <jamespage> so in maven.ignoreRules you can see the two plugins that we told mh_make to ignore - these get transformed out of the pom.xml during the build 84 [20:34] <jamespage> maven.rules is pretty simple in this case - this will create a 1.x version alongside the 1.2 version in /usr/share/maven-repo 85 [20:34] <jamespage> so we also get a maven.cleanIgnoreRules 86 [20:34] <jamespage> this is used when the clean target is called for the project 87 [20:35] <jamespage> typically this requires alot less dependencies so it may be a longer list of exclusions 88 [20:35] <jamespage> for this package its OK for it to be the same as maven.ignoreRules 89 [20:35] <jamespage> so I'll just do a quick change to the changelog and we are good to go 90 [20:36] <jamespage> You will notice that mh_make has used Java Demo <email@example.com> 91 [20:36] <jamespage> this gets picked up from DEBEMAIL and DEBFULLNAME environment variabled - I set these up earlier 92 [20:37] <jamespage> so source package build first 93 [20:38] <jamespage> maven-debian-helper patches/unpatches the pom.xml file during the process 94 [20:38] <jamespage> the default target for building the package is 'package' - however for more complex packages 'install' may be more appropriate. 95 [20:38] <jamespage> generating javadoc 96 [20:38] <jamespage> (twice - this is a bug!) 97 [20:39] <jamespage> automatically determining dependencies for the binary packages 98 [20:40] <jamespage> and done - a few lintian warnings but nothing unsolvable 99 [20:40] <jamespage> so we now have a built package \o/ 100 [20:41] <jamespage> as we discussed earlier the package deploys 1.2 and 1.x artefacts to /usr/share/maven-repo 101 [20:42] <jamespage> it also deploys to /usr/share/java - this is to support ant/javahelper based build and applications that require this library 102 [20:42] <jamespage> and also a -doc package containing the generated API 103 [20:42] <jamespage> just to prove that we have created something that at least installs 104 [20:43] <jamespage> Obviously this is a relatively simple example; 105 [20:43] <jamespage> however the concepts covered in this session can be used to build up complex dependency chains of packages to support large Java applications like Jenkins. 106 [20:43] <jamespage> Maven actually makes packaging for Ubuntu easier; 107 [20:44] <jamespage> because it has a standard way of expressing dependencies and a rich metadata model it is easier to automate the production of packaging 108 [20:44] <jamespage> packages that use ant are more challenging as they express dependencies in a million bespoke ways 109 [20:44] <jamespage> (unless they are using a dependency management system like ivy which can make things a little easier). 110 [20:45] <jamespage> Questions? 111 [20:46] <ClassBot> coalitians asked: so mavens central repository is never used ? 112 [20:46] <jamespage> good question - the local Debian repo in /usr/share/maven-repo is used and Maven is executed fully offline 113 [20:47] <jamespage> maven-debian-helper handles this as part of the packaging build rules 114 [20:47] <jamespage> so if you switch back to the demo you can see that the debian/rules file is very simple - all of the logic in encapsulated in cdbs classes 115 [20:48] <jamespage> Any more for any more? 116 [20:48] <jamespage> OK 117 [20:48] <jamespage> I hope that you have found this session useful; 118 [20:49] <jamespage> I normally hang out on #ubuntu-server and #ubuntu-java so if you have any more questions please feel free to ping me. 119 [20:49] <jamespage> Thankyou and goodnight!