== Dev Week -- Server oriented packages -- mathiaz -- Mon, Jan 25 == UTC {{{#!IRC [20:02] allright - let's keep the ball rolling [20:02] kees: thanks for your presentation [20:03] Hello everyone! [20:03] My name is Mathias Gug and am part of the Ubuntu Server team: [20:03] https://wiki.ubuntu.com/ServerTeam [20:03] During the next (and final) hour of today's Ubuntu Developer week I'll talk about Server related packages. [20:04] These are packages that usually provide services running in the background as daemon. They should be operating smoothly unattended by the system administrator. [20:04] I'll cover different topics relevant to server packages. I'll answer any questio [20:04] ns on the topic after I've covered it before moving on to the next one. [20:05] let's get started with the first topic: [20:05] Log files [20:06] The location of log files should be /var/log/package.log or /var/log/package/. [20:06] The latter version is more frequent for daemons as log files are essential for these programs and multiple logs files are usually created. [20:07] For example the apache2 package stores all its log files in /var/log/apache2/: [20:07] The access logs are available in one file while the error log is stored in another one. [20:08] The Filesystem Hierarchy Standard is followed by Debian and Ubuntu and has a section about the location about log files: [20:08] http://www.pathname.com/fhs/pub/fhs-2.3.html#VARLOGLOGFILESANDDIRECTORIES [20:08] Since daemons can generate a lot of information while running rotating the logs are important. [20:09] The package maintainer should provide a default log rotation policy during package installation. [20:09] Some daemons will take care of rotating their logs automatically. Most of them won't though. [20:10] The recommended way is to include a configuration file for logrorate. [20:10] Let's have a look at the apache2 default logrotate configuration: [20:10] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/apache2/lucid/annotate/head%3A/debian/logrotate [20:11] Note that apache2 is reloaded once log files have been rotated. This is a common operation for daemons as they tend to keep their log files opened and need to be told to reopen their log files once logrotate has rotated them. [20:12] The logrotate configuration file should be installed as /etc/logrotate.d/package-name: [20:12] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/apache2/lucid/annotate/head%3A/debian/rules#L265 [20:12] There is a debhelper command that can help automate this: dh_installlogrotate [20:12] The logrotate man page has more information about the options for log rotation. [20:13] Log directories are usually included in the package themselves. [20:13] For example, /var/log/apache2/ is part of the apache2-common package: === test is now known as Guest65958 [20:13] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/apache2/lucid/annotate/head%3A/debian/apache2.2-common.dirs [20:14] However they often need to be changed by maintainer scripts: [20:14] In the postinst, ownership and permissions are updated since daemons usually run as a non-root user: [20:14] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/apache2/lucid/annotate/head%3A/debian/apache2.2-common.postinst#L18 [20:14] Log files should also be deleted with the package is purged: [20:15] http://bazaar.launchpad.net/%7Eubuntu-branches/ubuntu/lucid/apache2/lucid/annotate/head%3A/debian/apache2.2-common.postrm#L8 [20:15] More information about log files can be found in the relevant section of the Debian policy: [20:15] http://www.debian.org/doc/debian-policy/ch-files.html#s10.8 [20:15] Any questions related to log files? [20:16] < toabctl_> mathiaz, QUESTION: but the apache2 package does not use the dh_installlogrotate command, right? [20:16] toabctl_: yes - that is correct [20:16] < n3rd> QUESTION: Is there a mechanism to replicate server configurations from one machine to another, helps in avoiding reinstalling and setting up all the parameters, for example a machine had [20:16] toabctl_: if it would, it would be done in the debian/rules file [20:16] apache configured with bug tracking system, SCM etc, the same needs to be setup in another machine so it would be easy if we could just replicate the same. [20:16] sorry... [20:17] n3rd: this is a more generic question - related to configuration management system [20:17] n3rd: I'd look into things like puppet, chef or cfengine [20:17] cjohnston: next [20:17] < toabctl_> QUESTION: does debhelper provide more function to take care for logfile handling? for example package purge.. [20:18] toabctl_: I don't think so [20:18] anything other questions related to log files before we move on? [20:19] That looks like it. :-) [20:19] nope - let's move on to the next topic then: [20:19] Configuration files [20:19] Dealing with configuration files should be guided by the following principles: [20:19] 1. Keep the user modification during upgrades [20:20] 2. Have a default working configuration after package install [20:20] Per the Debian Policy: [20:20] A configuration file """affects the operation of a program, or provides site- or host-specific information, or otherwise customizes the behavior of a program. Typically, configuration files are intended to be modified by the system administrator (if needed or desired) to conform to local policy or to provide more useful site-specific behavior.""" [20:20] http://www.debian.org/doc/debian-policy/ch-files.html#s-config-files [20:21] All configuration files must reside in /etc/. [20:21] Configuration files must only be removed on package *purge*, not on package *removal*. [20:21] Packages should not modify configuration files belonging to other packages. [20:22] In order to have a working configuration file by default the package can either ship a default configuration file or generate one during installation by asking questions to the user. [20:22] The first option should be used if possible. Shipping a default configuration file as part of the package itself under /etc/ is the easiest way. [20:22] Files will be considered as conffile by dpkg which will handle them for you during package upgrades. [20:23] This is appropriate only if it is possible to distribute a default version that will work for most installations, although some system administrators may choose to modify it. [20:23] This implies that the default version will be part of the package distribution, and must not be modified by the maintainer scripts during installation (or at any other time). [20:23] For example /etc/apache2/apache2.conf is a conffile: [20:24] http://packages.ubuntu.com/lucid/amd64/apache2.2-common/filelist [20:24] It's installed during the package build via a rule: [20:24] http://bazaar.launchpad.net/%7Eubuntu-branches/ubuntu/lucid/apache2/lucid/annotate/head%3A/debian/rules#L264 [20:24] The other solution to provide a default working configuration file is to generate a valid one during package installation. [20:25] Various maintainer scripts are used to ask questions to the end user (optional) and then create a configuration file. [20:25] They will also handle all configuration changes during package upgrade. [20:25] One option is to generate the configuration file directly in the postinst script as done by the openssh-server package: [20:25] http://bazaar.launchpad.net/%7Eubuntu-branches/ubuntu/lucid/openssh/lucid/annotate/head%3A/debian/openssh-server.postinst#L295 [20:26] Quite often this is combined with debconf questions. [20:26] The user is asked for information by the package .config script. The postinst script generates the configuration file according to the user answers. [20:27] Another great practice for configuration is the use of /etc/package.d/ include directories. [20:27] If the program supports configuration file inclusion I strongly recommend to ship a default file that include a /etc/package.d/ directory. [20:27] That often helps other packages to integrate with your program by dropping their own configuration file in the /etc/package.d/ directory. [20:27] For example apache2 includes a configuration directory in its default configuration file: [20:27] http://bazaar.launchpad.net/%7Eubuntu-branches/ubuntu/lucid/apache2/lucid/annotate/head%3A/debian/config-dir/apache2.conf#L232 [20:28] The apache2 package actually provides multiple .d directories: [20:28] A conf.d for generic options: [20:28] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/apache2/lucid/files/head%3A/debian/config-dir/conf.d/ [20:28] A modules.d (called mods-available) to enable specific modules: [20:28] http://bazaar.launchpad.net/%7Eubuntu-branches/ubuntu/lucid/apache2/lucid/files/head%3A/debian/config-dir/mods-available/ [20:29] Thanks to this directory packages providing a specific apache2 modules can just ship a configuration file in /etc/apache2/mods-available/. There is no need to edit a configuration file. [20:29] A sites.d (called sites-available) to manage virtual hosts: [20:29] http://bazaar.launchpad.net/%7Eubuntu-branches/ubuntu/lucid/apache2/lucid/files/head%3A/debian/config-dir/sites-available/ [20:29] This is more targeted at sysadmin rather than package maintainers. [20:30] The apache2 package is a great example to look at how a package can provide the infrastructure to help other packages integrated with it. [20:30] More detailed about configuration files handling can be found in the Debian policy: [20:30] http://www.debian.org/doc/debian-policy/ch-files.html#s-config-files [20:30] For more information about Debconf: [20:30] http://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html#bpp-config-mgmt [20:30] Any questions related to configuration files? [20:31] QUESTION: how to handle a file which is needed in the directory /usr/share/pyshared/testprogram/settings.py ? just add a symlink from /usr/share/pyshared/testprogram to /etc/testprogram/settings.py ? [20:32] toabctl_: yes - that usually how things should be done [20:32] toabctl_: all configuration files should be in /etc/ [20:32] < n3rd> Q: how can i avoid trailing slashes in conf files [20:33] toabctl_: if programs are not able to handle, that it's advised to use symlink from /etc/ to the expected location [20:33] n3rd: I'm not sure I understand your question [20:33] < n3rd> Q: why some refer to httpd.conf and some to apache.conf, how exactly they work? [20:34] n3rd: this is specific to how apache2 is working [20:34] n3rd: IIRC httpd.conf is around for historical reasons [20:34] n3rd: apache.conf should include httpd.conf [20:35] < n3rd> Question : instead of localhost/test, how can i configure apache2 to point localhost:port to test dir in /var/www [20:35] n3rd: that's a support question for apache2 - I won't address it here [20:35] n3rd: I'd suggest to ask the question in #ubuntu-server [20:36] I think that's all for this topic - let'sm ove on [20:36] next topic: Upstart job [20:36] The init script subsystem in Ubuntu has seen a lot of activity in the past releases as we're moving more and more to upstart jobs. [20:36] I'll give an overview on some features from upstart that are very helpful in managing daemons. [20:37] Upstart is able to supervise services: if a process dies Upstart can be configured to restart it. [20:37] For example, the openssh-server upstart job: [20:37] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/openssh/lucid/annotate/head%3A/debian/openssh-server.upstart [20:37] Using the respawn keywork instructs upstart to restart the sshd process if it disappears: [20:38] Line11 [20:38] Upstart also supports forking daemons and will supervise them correctly: [20:38] Line10 [20:38] Some daemons could also be run in the foreground by upstart: [20:38] http://bazaar.launchpad.net/%7Eubuntu-branches/ubuntu/lucid/eucalyptus/lucid/annotate/head%3A/debian/eucalyptus-cc.upstart#L56 [20:38] Another strength of Upstart is its dependency system. If a service is restarted other daemons may need to be restarted as well. [20:39] For example if the portmap service is restarted, gssd and statd will automatically be restarted by upstart as well: [20:39] http://bazaar.launchpad.net/%7Eubuntu-branches/ubuntu/lucid/nfs-utils/lucid/annotate/head%3A/debian/nfs-common.gssd.upstart#L9 [20:39] Upstart has built-in support for oom handling: [20:39] http://bazaar.launchpad.net/%7Eubuntu-branches/ubuntu/lucid/openssh/lucid/annotate/head%3A/debian/openssh-server.upstart#L14 [20:39] Other features include nicing and setting limits. See init(5) manpage for more information. [20:40] Traditionally daemon init scripts would source /etc/default/servicename. The default file provides an easy way to customize major options for the system administrator. [20:40] In Upstart it's suggested to modify directly the upstart job instead. [20:40] Upstart jobs should be defined in debian/package.upstart. They will be automatically handled by the relevant debhelper scripts (such dh_installinit). [20:41] More information can be found in the init(5) manpage and the Debian policy. [20:41] And don't forget the upstart website at: [20:41] http://upstart.ubuntu.com/ [20:42] Any question related to upstart? [20:43] < bullgard> QUESTION: What do you mean by "service" here? [20:43] bullgard: I think this has already been well answered in -chat [20:44] That's all I see [20:44] rmunn_: thanks [20:44] < zul> QUESTION: when would you use fork and when would you use daemon? [20:44] zul: not sure [20:45] < rmunn> QUESTION re upstart: How difficult is a typical init-to-upstart conversion? E.g., if I have a package using init scripts, is it usually worth the effort of rewriting its init scripts to use [20:45] upstart instead? How much is gained to offset the cost of the rewrite? [20:45] rmunn_: it depends on complex the init script is [20:46] rmunn_: usually it's worth the effort - as there is bunch of new features that upstart handles automatically for you [20:46] rmunn_: and if you do a lot of funky stuff in the init script (like modify configuration files), you can just use the same code in the pre/post-script stanzy in the upstart job [20:46] rmunn_: so quite often you can just copy and paste the code [20:47] all right - let's move on [20:47] Next topic: UFW profile [20:47] The security team wrote a simple host-based firewall tool named UFW: [20:47] https://wiki.ubuntu.com/UncomplicatedFirewall [20:47] Package integration is now available by dropping a file in /etc/ufw/applications.d/ to tell ufw how to configure firewalling rules for the services. [20:47] For example here is openssh-server ufw profile: [20:47] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/openssh/lucid/annotate/head%3A/debian/openssh-server.ufw.profile [20:48] And how to install it during the package build: [20:48] http://bazaar.launchpad.net/%7Eubuntu-branches/ubuntu/lucid/openssh/lucid/annotate/head%3A/debian/rules#L198 [20:48] For more information see the ufw wiki page at: [20:48] https://wiki.ubuntu.com/UncomplicatedFirewall [20:49] any question related to UFW? [20:50] None seen [20:50] allright then - let's move on [20:50] Next topic: AppArmor profile [20:50] Another tool that the security team is maintaining and the proves to be useful for services is AppArmor. [20:50] It can be seen as an alternative to chroots for daemons. [20:51] For example here is the ntp apparmor profile: [20:51] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/ntp/lucid/annotate/head%3A/debian/apparmor-profile [20:51] It should be installed in /etc/apparmor.d/path.to.daemon.binary: [20:51] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/ntp/lucid/annotate/head%3A/debian/rules#L72 [20:51] Some care needs to be taken during package upgrade especially when a new profile is introduced as we don't want to break existing systems. [20:52] More information about migrating a profile can be found on the wiki page: [20:52] https://wiki.ubuntu.com/ApparmorProfileMigration [20:52] And you can find more info on the AppArmor wiki page: [20:53] https://wiki.ubuntu.com/AppArmor [20:53] any question related to AppArmor profiles? [20:53] None seen [20:54] ok - let's move on [20:54] next topic: system users [20:54] One of the Ubuntu goal is to try to run as many daemons as possible as non-root users. [20:54] That brings the need to create system users. The best practice is to not delete users on package purge. Otherwise the uid could be reallocated, potentially giving access to files from the previous package to the new program. [20:55] For example the openssh-server postinst creates the ssshd user if the user doesn't already exists: [20:55] http://bazaar.launchpad.net/%7Eubuntu-branches/ubuntu/lucid/openssh/lucid/annotate/head%3A/debian/openssh-server.postinst#L401 [20:55] If the daemon requires to be run as root try to write an AppArmor profile for it. [20:56] More information about system users can be found in the Debian policy: [20:56] http://www.debian.org/doc/debian-policy/ch-opersys.html#s9.2 [20:56] Any question related to system users? [20:57] nope - let's move on then [20:57] Next topic: Cron jobs [20:57] Cron jobs should installed in /etc/cron.d/ or any of the regular cron directory (ex: /etc/cron.daily/). [20:58] There is a debhelper script available for packager: dh_installcron. [20:58] The Debian policy has more information: [20:58] http://www.debian.org/doc/debian-policy/ch-opersys.html#s9.5 [20:58] Any questions related to cron jobs? [20:59] None seen... [20:59] allrigh then [20:59] that's all for now then [20:59] you can find more information in the following guides: [20:59] * Debian policy: - http://www.debian.org/doc/debian-policy/ - debian-policy package [20:59] * Debian Developer's Reference: - http://www.debian.org/doc/manuals/developers-reference/index.en.html [20:59] * Debian New Maintainers' Guide: - http://www.debian.org/doc/manuals/maint-guide/index.en.html [20:59] * Ubuntu Policy: - ubuntu-policy package [21:00] * Ubuntu Package Guide: - https://wiki.ubuntu.com/PackagingGuide/ [21:00] * Mailing lists and IRC channels: - ubuntu-devel@, ubuntu-server@ - #ubuntu-devel, #ubuntu-server [21:00] * Ubuntu wiki: - https://wiki.ubuntu.com [21:00] And one last page that I often use when I have my packager hat: [21:00] http://wiki.debian.org/MaintainerScripts [21:00] this one outlines the sequence in which maintainer scripts are called [21:01] and with that the first day of the Ubuntu Developer Week is ending [21:01] come back for more tomorrow, same place at 16:00 UTC [21:02] you'll learn about java libraries, ubuntu one support, automated server testing and other things [21:02] https://wiki.ubuntu.com/UbuntuDeveloperWeek/ [21:02] thanks all and bye! }}}