Introduction

This document compares Upstart and systemd with a view to aiding in the transition to the latter.

Support status

First, it is important to note that systemd is only fully supported in Ubuntu 15.04 and later releases. While systemd is available in prior releases through the Ubuntu repositories, there is a deemphasis of support for these releases as noted here. Hence, it is advised to use the default upstart on prior releases.

System Init Daemon

This has changed as part of the Ubuntu 15.04 devel cycle.

Ubuntu 15.04 (using Systemd by default):

• Systemd runs with PID 1 as /sbin/init.

• Upstart runs with PID 1 as /sbin/upstart.

Prior versions (using Upstart by default):

• Upstart runs with PID 1 as /sbin/init.

• Systemd runs with PID 1 as /lib/systemd/systemd.

Switching init systems

If you are running Ubuntu vivid (15.04), you can easily switch between upstart and systemd at will since both packages are installed at present. As of March 9 2015, vivid was changed to use systemd by default, before that upstart was the default.

Switch to upstart for a single boot

In grub, select "Advanced options for Ubuntu", where you will find an "Ubuntu, with Linux ... (upstart)" entry. This will boot with init=/sbin/upstart.

If you have upstart-sysv installed and thus boot with upstart by default, there will be an "Ubuntu, with Linux ... (systemd)" entry, which will boot with init=/lib/systemd/systemd.

Permanent switch back to upstart

Install the upstart-sysv package, which will remove ubuntu-standard and systemd-sysv (but should not remove anything else -- if it does, yell!), and run sudo update-initramfs -u. After that, grub's "Advanced options" menu will have a corresponding "Ubuntu, with Linux ... (systemd)" entry where you can do an one-time boot with systemd.

If you want to switch back to systemd, install the systemd-sysv and ubuntu-standard packages.

High-level startup concept

Upstart's model for starting processes (jobs) is "greedy event-based", i. e. all available jobs whose startup events happen are started as early as possible. During boot, upstart synthesizes some initial events like startup or rcS as the "tree root", the early services start on those, and later services start when the former are running. A new job merely needs to install its configuration file into /etc/init/ to become active.

systemd's model for starting processes (units) is "lazy dependency-based", i. e. a unit will only start if and when some other starting unit depends on it. During boot, systemd starts a "root unit" (default.target, can be overridden in grub), which then transitively expands and starts its dependencies. A new unit needs to add itself as a dependency of a unit of the boot sequence (commonly multi-user.target) in order to become active.

Job vs. unit keywords

This maps the keywords that can occur in an upstart job to the corresponding ones in a systemd unit. Keywords which don't have a direct equivalent are marked with "-".

Shell scripts

systemd does not provide special support for shell scripts (by design). For short shell commands you can use something like

   ExecStart=/bin/sh -ec 'echo hello'

Longer scripts are usually program logic and should not be directly in a conffile and duplicated between upstart and systemd; factor it out in a proper script in e. g. /usr/share/myapp/ and call it from both the upstart job and the systemd unit.

Automatic starting

As described above, services which want to start during boot (i. e. are not activated through sockets, D-BUS, or similar) need to become a dependency of an existing boot target. Those need an [Install] section with a WantedBy= that specifies the unit which that new service wants to become a dependency of (see man systemd.unit). Very commonly this is multi-user.target, which is roughly equivalent to start on runlevel [2345] in upstart; see man systemd.special for other common targets.

Invalid configurations

• upstart
  • Refuses to start a job if it contains invalid syntax.
• systemd

  • Ignores invalid directives and starts the service. Note that you must test your configuration carefully since a typo will not be detected! Use systemd-analyze verify <unit> file to get warnings on typos and badly formatted options.

Override Files

• upstart

  • Upstart allows part or all of a Job Configuration ("/etc/init/$job.conf") file to be overridden. To create an override file:

  • Create file "/etc/init/$job.override" containing one or more stanzas which take priority over their counterparts from the original "$job.conf" file.

• systemd
  • Systemd allows a similar facility via "drop-ins": a drop-in allows a directive to be modified without changing the original unit file. To create a drop-in:

  • Create a subdirectory below either "/etc/systemd/system/" or "/lib/systemd/system/" called "${unit}.d/".

  • Create files called <something>.conf in the "${unit}.d/" directory containing the directives that you wish to override.

Behavioral differences (aka Gotchas)

• systemd will refuse to run a binary (via ExecStart, ExecStartPre, etc) unless the full path to the binary is specified:

  • BAD (systemd will not search for sleep in $PATH):

       ExecStart=sleep 20

    GOOD (absolute path specified):

       ExecStart=/bin/sleep 20

Commands

Note that these are commands for interactive human usage. Package maintainer scripts, ifupdown hooks and similar must always use the init system agnostic abstractions like invoke-rc.d.

Example Services

Example Upstart Service

/etc/init/foo.conf:

description "Job that runs the foo daemon"

# start in normal runlevels when disks are mounted and networking is available
start on runlevel [2345]

# stop on shutdown/halt, single-user mode and reboot
stop on runlevel [016]

env statedir=/var/cache/foo

# create a directory needed by the daemon
pre-start exec mkdir -p "$statedir"

exec /usr/bin/foo-daemon --arg1 "hello world" --statedir "$statedir"

Example Systemd service

/lib/systemd/system/foo.service:

[Unit]
Description=Job that runs the foo daemon
Documentation=man:foo(1)

[Service]
Type=forking
Environment=statedir=/var/cache/foo
ExecStartPre=/usr/bin/mkdir -p ${statedir}
ExecStart=/usr/bin/foo-daemon --arg1 "hello world" --statedir ${statedir}

[Install]
WantedBy=multi-user.target

Outstanding Work

If you'd like to help out with the migration, take a look at ...

• Outstanding packages to convert:

  • http://people.canonical.com/~jhunt/systemd/packages-to-convert/

    • Note that the priority are the packages in "main".

• blueprint: https://blueprints.launchpad.net/ubuntu/+spec/core-1411-systemd-migration

... and then come and chat to us on #ubuntu-devel.

Common Idioms

Service that specifies shell meta-characters

• upstart

  • Upstart automatically detects if a job requires a shell to expand meta characters. For example, the following job will automatically be run via "/bin/sh -e":

       exec mydaemon --date $(date)
       

• systemd
  • systemd does no such detection so the equivalent unit file would need to be:

       [Service]
       ExecStart=/bin/sh -ec "exec /usr/bin/mydaemon --date $(date)"
       

Service that does not fork (runs in foreground)

• upstart job

  •    description "blah, blah, blah"
       exec sleep 999
       

• systemd unit

  •    [Unit]
       Description=blah, blah, blah
       
       [Service]
       Type=simple
       # (NOTE: "Type=simple" is the default)
       ExecStart=/usr/bin/sleep 999
       

Do not run service if no daemon configuration file created

• upstart job

  •    # defines PORT variable
       env config=/etc/default/foo
    
       pre-start script
       [ -e "$config" ] || { stop; exit 0; }
       end script
    
       exec mydaemon --port="$PORT"
       

• systemd unit

  •    [Service]
       Type=forking
       EnvironmentFile=/etc/default/foo
       ExecStart=/usr/bin/mydaemon --port=$PORT
       

Note: EnvironmentFile=/etc/default/foo is making /etc/default/foo mandatory. If the environment file (and so variables) are optional, you can use: EnvironmentFile=-/etc/default/foo (notice the - in front of the path).

/etc/default files which enable/disable jobs

enable=1|0 type settings in /etc/default files should generally be avoided. The canonical way to enable/disable a service in an init system agnostic way is update-rc.d <service> enable|disable, which will translate to init system specific actions such as adding/removing symlinks (SysV and systemd) or creating/removing job override files (upstart). For systemd in particular, admins also often call systemctl enable|disable <service> directly. Thus these settings are redundant in /etc/default.

There is no clean way to evaluate these in a systemd unit. You can check them in ExecStartPre=, but that would mean that the unit will be in "failed" state if the service gets disabled that way, and so, is not desirable.

For these reasons (confusing/duplication/cannot be modelled in systemd), these settings should be removed. This was done in whoopsie 0.2.42, you can check its diff for a transition which respects the old default setting and removes it on upgrade.

Start a service when a file is created

• upstart job

  •    start on file FILE=/var/crash/*.crash EVENT=create
       exec crash-handler-daemon
       

• systemd
  • Two files are required:

  • /lib/systemd/system/foo.service:

       [Service]
       Type=forking
       ExecStart=/usr/bin/crash-handler-daemon
       

  • /lib/systemd/system/foo.path:

       [Path]
       PathExistsGlob=/var/crash/*.crash
       Unit=foo.service
       # (NOTE: the default is "Unit=foo.service" if this is called "foo.path")
       

Start a service when a D-Bus name is acquired

• upstart job

  •    start on dbus SIGNAL=NameAcquired INTERFACE=... OBJPATH=... SENDER=... DESTINATION=...
       exec mydaemon
       

• systemd

  • /lib/systemd/system/foo.service:

       [Unit]
       Description=Service that acquires a D-Bus name
       
       [Service]
       Type=dbus
       # (NOTE: systemd will consider this service ready when it claims the 'foo.bar.baz' name on dbus)
       BusName=foo.bar.baz
       ExecStart=/usr/bin/mydaemon
       

  • /lib/systemd/system/bar.service:

       [Unit]
       Description=Service that needs the foo service
       Requires=foo.service
       After=foo.service
       
       [Service]
       ExecStart=/usr/bin/mydaemon2
       

Well-known Sequence Points

• Upstart provides a set of "well-known" events that jobs can make use of (such as "runlevel" and "started"). These are summarised in the upstart-events(7) manual page.

• Systemd has a similar concept in the form of "targets". See the systemd.special(7) manual page for details and bootup(7) for the boot sequence.

• For the particular case of upstart's static-network-up: Our ifupdown integrates that into network-online.target (see man systemd.special(7)). Thus, translate start on static-network-up to Requires/After=network-online.target.

Environment Differences

Both Upstart and systemd provide a very limited environment to the services they run. The environment can be modified by specifying additional stanzas (for Upstart) or directives (for systemd).

However, there are some subtle differences between the default service environment provided by both init systems. The table below shows the major differences:

Note that this information is subject to change; to determine a precise difference on your system:

1. Install the procenv tool:

    $ sudo apt-get -y install procenv
    

2. Create an Upstart job to run procenv:

    $ cat <<EOT | sudo tee /etc/init/procenv.conf
    description "Display Upstart environment"
    exec procenv --file=/tmp/procenv-upstart.log
    EOT
    

3. Run procenv under upstart:

    $ sudo start procenv
    

4. Create a systemd unit file to run procenv:

    $ cat <<EOT | sudo tee /lib/systemd/system/procenv.service
    [Unit]
    Description=Display systemd environment
   
    [Service]
    Type=oneshot
    ExecStart=/usr/bin/procenv --file=/tmp/procenv-systemd.log
    EOT
    

5. Run procenv under systemd:

    $ sudo systemctl start procenv
    

6. Compare the environments:

    $ diff /tmp/procenv-upstart.log /tmp/procenv-systemd.log
    

Common Problems

How to identify which init system you are currently booting with

At the time of writing, it is possible to boot an Ubuntu system with either Upstart or systemd since both are installed.

To determine which init daemon you are currently booting with, run:

  $ ps -p1 | grep systemd && echo systemd || echo upstart

Why don't some commands (like grep) work in /lib/systemd/system ?

If you hit this problem...

  $ cd /lib/systemd/system
  $ grep foo *
  grep: invalid option -- '.'
  Usage: grep [OPTION]... PATTERN [FILE]...
  Try 'grep --help' for more information.

... you need to add "--" to the grep(1) call:

  $ cd /lib/systemd/system
  $ grep foo -- *

This is required since systemd provides a file called "-.slice". After the shell has expanded the asterisk ("*"), that file gets passed to the command in question (grep, cat, etc) and that command will then attempt to interpret "-.slice" as a command-line option rather than a filename.

Alternative work-around:

  $ cd /lib/systemd/system
  $ POSIXLY_CORRECT=1 grep foo *

To view the pesky file:

  $ cd /lib/systemd/system
  $ cat -- -.slice

Note that this issue only occurs when your current working directory is the directory containing "-.slice". The simplest work-around is to make the grep or cat call from a different directory:

  $ grep foo /lib/systemd/system/*
  $ cat /lib/systemd/system/-.slice

Debugging

Since this implies the service works under Upstart but is problematic under systemd, details of both systems are provided to allow for some comparison.

Boot Time

Common Setup

Remove the following from the kernel command-line via the grub menu:

• "quiet"

• "splash"

Upstart

• Add "--debug to the kernel command-line via the grub menu.

• Optionally add "console=ttyS0 to the kernel command-line via the grub menu if you have a serial console.

systemd

• Add "systemd.log_level=debug  to the kernel command-line via the grub menu.

• Optionally add one of the following too:

  • "systemd.log_target=kmsg"

  • "systemd.log_target=console"

Starting a rescue shell

• Run:

  •    $ sudo systemctl enable debug-shell.service

• Reboot.
• If the system fails to boot, you can now switch to tty9 (CTRL+ALT+F9) for a getty console login.

From a running system

Upstart

To switch to debug mode for the system init (PID 1):

  $ sudo initctl log-priority debug

To switch to debug mode for a session init (init PID != 1):

  $ initctl log-priority debug
  $ tail -f ~/.xsession-errors

Debian Packaging

Packages installing systemd services should build-depend on dh-systemd and either call dh --with systemd (if they use dh) or call dh_systemd_enable and dh_systemd_start before/after dh_installinit respectively. Files under debian called *.service will be installed analogously to *.upstart files. See dh_systemd_enable(1p) for how to customize the installation.

Further Information

• man systemd -- "CONCEPTS" has an useful link collection to the per-topic pages

• Ubuntu wiki systemd page (not relevant for porting)

• Upstream systemd page with lots of documentation pointers

• man 8 init, man 5 init (on an Upstart system)

• systemd manpages for service writers:

  • http://www.freedesktop.org/software/systemd/man/systemd.unit.html

  • http://www.freedesktop.org/software/systemd/man/systemd.service.html

  • http://www.freedesktop.org/software/systemd/man/systemd.exec.html

  • http://www.freedesktop.org/software/systemd/man/systemd.special.html

• Upstart

  • Upstart Cookbook