Ubuntu Wiki

Dev Week -- Authoring upstart jobs -- SteveLangasek -- Mon, Jul 12th, 2010

(04:00:33 PM) slangasek: hey folks
(04:00:57 PM) akgraner: hey Steve if you are ready the floor is yours!
(04:02:43 PM) slangasek: in this session, I'd like to cover the basics of writing upstart jobs, cover a few examples, look at a few special cases of interest, and then open the floor to questions
(04:03:07 PM) slangasek: if you have questions as we go, please ask (the usual method on #ubuntu-classroom-chat)
(04:04:07 PM) slangasek: so as written at https://wiki.ubuntu.com/UbuntuDeveloperWeek/Sessions, it is strongly recommended that you read over the init(5) manpage first, to get an understanding of what the various commands are that you can use in an upstart job
(04:05:06 PM) slangasek: so I won't repeat here the contents of that manpage; that's a bit too much to cover in detail in a 1h session
(04:05:29 PM) slangasek: but if you haven't read it yet, you may still learn something of interest here and can always go read the full details later!
(04:05:48 PM) slangasek: but first, a rhetorical question: why would you want to write an upstart job?
(04:05:57 PM) slangasek: Ubuntu uses upstart as its init daemon
(04:06:53 PM) slangasek: if you want to create a new service on Ubuntu - like a mail daemon, web server, or system audio daemon, for example - the recommended way to do this is by writing an upstart job
(04:07:23 PM) slangasek: even running arbitrary commands at startup or shutdown, that have nothing to do with a service, can be implemented as upstart jobs
(04:07:43 PM) slangasek: upstart jobs supersede the traditional sysvinit rc system of starting and stopping jobs (those files under /etc/init.d/)
(04:08:21 PM) slangasek: both are still supported - as explained at https://help.ubuntu.com/community/UpstartHowto - but for new services, it's definitely recommended to use native upstart jobs for several reasons
(04:09:06 PM) slangasek: - upstart is a process supervisor in addition to being an init daemon, so it always knows the state of the job and can handle start/stop directly and respawn services when they die
(04:09:17 PM) slangasek: (so no messy bolted-on pid file handling!)
(04:09:46 PM) slangasek: - upstart jobs can be declared to start and stop on a number of different conditions, not just on runlevel changes
(04:10:27 PM) slangasek: - upstart job syntax is much simpler than sysv init scripts - you *can* write complicated scripts inside of an upstart job, but for the common case, there's much less scripting involved!
(04:12:15 PM) slangasek: now let's start with a simple example of an upstart job.  Since you might not have all of these packages installed on your system, I've copied them to the web for convenience: http://people.canonical.com/~vorlon/upstart/
(04:12:22 PM) slangasek: the first once we'll look at is http://people.canonical.com/~vorlon/upstart/acpid.conf
(04:12:44 PM) slangasek: this is a bog-standard upstart job, nothing too complicated at all
(04:12:54 PM) slangasek: it has a description, which is optional but used by upstart in some informational messages
(04:13:08 PM) slangasek: two lines tell us when to start and when to stop the service
(04:13:22 PM) jorge is now known as jcastro
(04:13:59 PM) slangasek: here, the start and stop conditions relate in a pretty straightforward manner to the sysvinit runlevels - start the service on any of runlevels 2, 3, 4, or 5; and stop it on any runlevel *not* in that list
(04:15:03 PM) slangasek: (which, if you know your runlevels, means to stop it on runlevels S, 0, 1, or 6: i.e., on shutdown, reboot, single user mode, and runlevel 1 which is a sort of "faux single user mode"
(04:15:07 PM) slangasek: )
(04:16:44 PM) slangasek: after that we have the line 'expect fork', which tells us how the service starts up in order for upstart to be able to supervise the process correctly, know which process is the master process and when it's done starting up so it can trigger other jobs that depend on it, etc
(04:16:52 PM) slangasek: this line is *very important* to get right
(04:17:16 PM) slangasek: if you get it wrong, you may not be able to fix it without rebooting your computer!
(04:17:20 PM) slangasek: more on that later :)
(04:17:44 PM) slangasek: the next line is 'respawn', which tells upstart that if the job stops, it should be restarted
(04:18:02 PM) slangasek: there are a number of different options you can use to configure how often upstart will respawn the service
(04:18:09 PM) slangasek: all documented in the glorious manpage
(04:18:23 PM) slangasek: finally, we have a single line describing what the job itself *does*
(04:18:25 PM) slangasek: exec acpid -c /etc/acpi/events -s /var/run/acpid.socket
(04:19:11 PM) slangasek: the 'exec' tells upstart to run the following command directly
(04:19:16 PM) slangasek: you could also write this as:
(04:19:17 PM) slangasek: script
(04:19:25 PM) slangasek:    exec acpid -c /etc/acpi/events -s /var/run/acpid.socket
(04:19:26 PM) slangasek: end script
(04:19:43 PM) slangasek: ... but there's not much point to doing that, the 'script' and 'end script' lines are redundant
(04:19:52 PM) slangasek: also, note that this is *not* the same:
(04:19:55 PM) slangasek: script
(04:19:59 PM) slangasek:    acpid -c /etc/acpi/events -s /var/run/acpid.socket
(04:20:00 PM) slangasek: end script
(04:21:16 PM) slangasek: it's not the same because when you close the job in 'script', upstart spawns a shell; the shell is a process, which spawns a subprocess when you call acpid by itself, and that confuses upstart because we told it that the process is done starting up as soon as there's a subprocess (expect fork)!
(04:22:29 PM) slangasek: looks like there are a couple of good questions queued up, let's get to those before we go on
(04:23:02 PM) slangasek: please remember to type 'QUESTION: ' in front of your question, not just 'QUESTION' so the bot can parse it - that'll help us go quicker :)
(04:23:08 PM) ClassBot: Omahn18 asked: Does upstart have a toggle switch that can be enabled to log all service transistions to /var/log/daemon or equivalent for debugging purposes?
(04:23:55 PM) slangasek: good question - yes, you can make upstart more verbose by running 'sudo initctl log-priority info' or 'sudo initctl log-priority debug'
(04:24:27 PM) slangasek: 'info' is normally enough for debugging any problems you might have with new jobs you've written - 'debug' is usually way too verbose for anything other than debugging upstart itself :)
(04:25:25 PM) slangasek: and if you need to debug upstart jobs that happen too early in the boot sequence before you're able to *type* 'sudo initctl [...]', you can actually just type '--verbose' or '--debug' on the kernel commandline - upstart will parse that and go into that mode on startup :)
(04:25:59 PM) ClassBot: Ram_Yash asked: Whenever I upstart the music player it disappers first time without any processor . Is this is a bug or Am I missing something?
(04:26:46 PM) slangasek: Ram_Yash: have you written an upstart job for this?  If so, please post it somewhere that we can look at it (e.g., pastebin.ubuntu.com) and we'll discuss it a little later
(04:27:05 PM) slangasek: if you mean "when you start up the music player", then that doesn't have anything to do with upstart jobs...
(04:27:18 PM) ClassBot: simar_mohaar asked: Where to place these scripts so that they run automatically ...
(04:27:31 PM) slangasek: upstart finds its jobs in /etc/init
(04:27:52 PM) slangasek: adding a job file there with a name ending in .conf is enough to get upstart to see it and act on it
(04:28:39 PM) slangasek: (but it won't automatically start it for you, because start conditions are defined by events and upstart won't automatically rerun the events for you - so you would need to run 'sudo service $job start' to start it)
(04:30:00 PM) slangasek: some more good questions coming in about upstart job authoring - let's go to a couple more examples and see if we can't answer those questions along the way
(04:30:18 PM) slangasek: http://people.canonical.com/~vorlon/upstart/upstart-udev-bridge.conf
(04:30:39 PM) slangasek: just a couple of interesting differences here from the acpid job
(04:30:52 PM) slangasek: instead of 'start on runlevel [...]', it's 'start on starting udev' and 'stop on stopped udev'
(04:31:22 PM) slangasek: these events are internal events, documented in the init(5) manpage: whenever an upstart job starts or stops, events are sent that other jobs can trigger on
(04:31:36 PM) slangasek: so this way, we can say "upstart-udev-bridge should only run when udev is running"
(04:31:58 PM) slangasek: now, 'runlevel' is also an event, but it's not documented in the manpage because it's not an event that's internal to upstart
(04:32:27 PM) slangasek: er
(04:32:44 PM) slangasek: sorry, that's incorrect - that event *is* internal to upstart, it's emitted whenever you change runlevels :)
(04:32:55 PM) slangasek: the other thing that's different here is 'expect daemon' instead of 'expect fork'
(04:33:23 PM) slangasek: 'expect daemon' means we expect the service to daemonize (in a nutshell: to fork twice) to let us know that it's ready to go
(04:33:38 PM) slangasek: if you *can* use 'expect daemon', that's generally recommended for upstart jobs related to services
(04:33:53 PM) slangasek: but sometimes it's not possible because the behavior of the daemon when daemonizing isn't what upstart expects
(04:34:01 PM) slangasek: e.g., http://people.canonical.com/~vorlon/upstart/smbd.conf
(04:34:47 PM) slangasek: smbd, from the samba package, daemonizes by default... but we have to tell it to run in the foreground here (exec smbd -F) and *not* use an 'expect fork' or 'expect daemon' line, because smbd manages to confuse upstart badly enough that it hangs the job and you have to restart to continue debugging ;0
(04:34:53 PM) slangasek: so watch out for that :)
(04:34:59 PM) slangasek: another new thing in here is a pre-start script
(04:35:27 PM) slangasek: upstart jobs can be told to run commands, or entire scripts, before and after starting, and before and after *stopping*, the main process
(04:35:49 PM) slangasek: you can use any combination of these that you need to
(04:36:01 PM) slangasek: and should leave out the ones that you don't. :)
(04:36:06 PM) slangasek: that brings us to another question....
(04:36:09 PM) ClassBot: Omahn18 asked: I've written an upstart job for NIS (bug 569757) and ideally it should trigger a restart of autofs (if installed) to pull down the NIS automount maps. Does upstart have any specific functionality to do this or should it just be included in the script section?
(04:36:47 PM) slangasek: yep - just put a command in your post-start script to trigger the reload
(04:37:07 PM) slangasek: if autofs is an upstart job itself, you can just run 'restart autofs'
(04:37:32 PM) slangasek: but I would recommend thinking carefully about whether this is the right thing to do... should autofs be restarted, or just sent a signal to get it to reload its configuration?
(04:37:57 PM) slangasek: if the latter, you can get the pid of the job you want to restart by running 'status autofs'
(04:39:33 PM) slangasek: well, actually... as long as the service responds to SIGHUP, you can run 'reload autofs'
(04:39:54 PM) ClassBot: tech2077 asked: QUESTION: while looking at the files in /etc/init.d/, they are all system v init daemons, not upstart daemons, are there any ones in teh folder by standard install that are
(04:40:24 PM) slangasek: upstart jobs are all in /etc/init, not /etc/init.d.  We will probably eventually get rid of /etc/init.d on a default system, but we're still transitioning...
(04:40:32 PM) ClassBot: simar_mohaar asked: Can i see which all upstart jobs are currently running ?
(04:40:50 PM) slangasek: great question!  yes, you can run 'sudo initctl list'
(04:41:05 PM) slangasek: (if you're not root, you can't see the list - you can only check the status of individual jobs with 'status')
(04:41:19 PM) ClassBot: simar_mohaar asked: 'sudo service $job start' to start it  do i need to do it once or every time i start computer ??
(04:41:49 PM) slangasek: so earlier I mentioned that upstart won't automatically start a job for you when you create it
(04:42:09 PM) slangasek: but, if *after* creating it, the start condition is met, yes - it will be started for you
(04:42:39 PM) slangasek: so as long as you set the start condition to something that happens on startup, you don't need to run 'sudo service $job start' again after reboot
(04:42:52 PM) slangasek: here's an example of a start condition you can use on startup: http://people.canonical.com/~vorlon/upstart/rsyslog.conf
(04:43:00 PM) slangasek: 'start on filesystem'
(04:43:14 PM) slangasek: now this one is *not* an upstart built-in event
(04:43:17 PM) slangasek: (I'm sure this time :)
(04:43:23 PM) slangasek: it's an event that comes from mountall
(04:43:37 PM) slangasek: mountall gives us several events letting us know when the filesystem is mounted
(04:44:14 PM) slangasek: 'local-filesystems', 'filesystem', 'virtual-filesystems', 'remote-filesystems'... separate events for each separate filesystem that gets mounted
(04:44:24 PM) slangasek: now, generally the only one you want to use is 'filesystem'
(04:44:46 PM) slangasek: for most jobs, you want to wait for *all* the filesystems in /etc/fstab to be mounted before you start your job
(04:44:58 PM) slangasek: starting it any earlier is probably just going to cause contention and slow the system down
(04:45:13 PM) slangasek: note that 'start on filesystem' means that your job will start *regardless* of what runlevel you're booting into
(04:45:26 PM) slangasek: so even in single user mode, rsyslog is still started
(04:45:42 PM) slangasek: and it only stops on shutdown or reboot
(04:46:01 PM) slangasek: so if your service doesn't really belong in single user mode, you should use 'start on runlevel [2345]' instead
(04:46:20 PM) slangasek: the other mountall signal that's worth mentioning, that you may see in a lot of other jobs, is 'local-filesystems'
(04:46:31 PM) slangasek: in fact, we already saw one of these: smbd uses it
(04:46:56 PM) slangasek: 'local-filesystems' is different from 'filesystem' in that it's emitted before your remote filesystems are up
(04:47:27 PM) slangasek: so you would use 'start on local-filesystems' for *only* those jobs that may be needed in order to get your remote filesystems mounted: networking, filesystem daemons, etc
(04:47:57 PM) slangasek: otherwise, the files your job needs to run may be *on* a remote filesystem, and it'll try to start it too early and fail!
(04:48:32 PM) slangasek: Moomoc asked: 'Is it possible to use start on file-system [23] e.g.?'
(04:48:45 PM) slangasek: well, there's no event named 'file-system 2'
(04:48:59 PM) slangasek: there is a 'mounted' and a 'mounting' event emitted for each mount point, when it's mounted
(04:49:16 PM) slangasek: *but*, that means the start condition depends on the partition layout of the system!
(04:49:32 PM) slangasek: so you don't want to do that for any jobs that go into Ubuntu itself
(04:49:42 PM) ClassBot: simar_mohaar asked: What are upstart jobs commonly used for??
(04:50:05 PM) slangasek: upstart jobs are used for *everything* to do with starting up your system or shutting it down
(04:50:12 PM) slangasek: (with a few exceptions that haven't transitioned yet)
(04:50:22 PM) slangasek: mounting filesystems is done by an upstart job
(04:50:26 PM) slangasek: starting the X server
(04:50:37 PM) slangasek: starting dbus, avahi, cups, ...
(04:50:45 PM) slangasek: your clock settings
(04:50:48 PM) slangasek: your firewall
(04:50:49 PM) slangasek: etc
(04:50:56 PM) slangasek: there's a lot of stuff in /etc/init :)
(04:51:12 PM) slangasek: speaking of the firewall...
(04:51:15 PM) slangasek: http://people.canonical.com/~vorlon/upstart/ufw.conf
(04:51:26 PM) slangasek: that's how we start the ufw firewall on Ubuntu
(04:51:36 PM) slangasek: notice an interesting thing about this job?
(04:51:46 PM) slangasek: there's a pre-start exec and a post-stop exec, but there's no exec!
(04:51:54 PM) slangasek: even the main process part of an upstart job is optional
(04:52:06 PM) slangasek: you might be wondering why you would want to do that, though
(04:52:38 PM) slangasek: it's because we want to know whether the firewall is stopped or started, but there's no daemon we run that's associated with it - it's all in the kernel
(04:53:04 PM) slangasek: so instead of making it a 'task' (which upstart would show as stopped once it's finished), we make it a regular service that runs nothing!
(04:53:12 PM) slangasek: this also prevents upstart from trying to start it repeatedly
(04:53:18 PM) slangasek: notice that the start condition here is:
(04:53:21 PM) slangasek: start on (starting network-interface
(04:53:22 PM) slangasek:           or starting network-manager
(04:53:22 PM) slangasek:           or starting networking)
(04:53:32 PM) slangasek: that tells upstart to start it whenever it sees *any* of these events
(04:53:47 PM) slangasek: which, since this is a service, means it will start it on the *first* of these events that it sees
(04:53:51 PM) slangasek: and then stay "runninG"
(04:54:24 PM) slangasek: I have more examples, but no time to go over them... let's go to questions :)
(04:54:33 PM) ClassBot: penguin42 asked: What should I ask someone to do to help debug a startup issue on a remote machine with upstart?
(04:55:18 PM) slangasek: penguin42: have them turn on the verbosity of upstart with 'sudo initctl log-priority info' and get them to send you /var/log/syslog - and then get good at understanding that output, by practicing on your own system :)
(04:55:24 PM) ClassBot: tech2077 asked: Can any command be exicuted as a upstart, or are there limitations to this system
(04:55:47 PM) slangasek: well, it's kind of a limitation that upstart jobs are "system-level" services
(04:55:59 PM) slangasek: so you wouldn't really want to use it to start per-user services
(04:56:04 PM) slangasek: but otherwise, there aren't many limitations
(04:56:28 PM) slangasek: if you have questions and you haven't asked them yet, please get them to the bot now :-)
(04:56:36 PM) slangasek: I'll be available afterwards to answer questions, too
(04:56:43 PM) ClassBot: ktenney asked: where is a list of all such external events available?
(04:56:46 PM) slangasek: good question
(04:56:57 PM) slangasek: unfortunately, there is no comprehensive list of external events
(04:57:04 PM) slangasek: because you can make up event names whenever you want!
(04:57:20 PM) slangasek: but many of them are documented in the 'emits' lines in other upstart jobs (grep -r emits /etc/init)
(04:58:04 PM) slangasek: some of them also come from ifupdown helpers under /etc/network/if-up.d (grep -r initctl /etc/network)
(04:58:15 PM) slangasek: but there's no master list, and yes, this is a problem :/
(04:58:28 PM) ClassBot: Ram_Yash asked: Can use UPSTART job to run web application?
(04:58:34 PM) slangasek: if the web application runs as its own process, yes
(04:58:51 PM) slangasek: though usually, web applications run inside the webserver; in that case, the upstart job would be the webserver itself
(04:59:04 PM) ClassBot: zul asked: what if you have a daemon that depends on loopback
(04:59:16 PM) slangasek: great illustrative question, though I think zul maybe already knows the answer :)
(04:59:50 PM) slangasek: rc-sysinit.conf is an example of this!
(04:59:58 PM) ClassBot: Omahn18 asked: Who would be the best person(s) to contact to assist in moving init.d scripts to upstart scripts?
(05:00:19 PM) slangasek: in general, the Ubuntu foundations team - and now the Ubuntu server team - have the most experience with this, and I'm sure are happy to help with your questions