AutomatedServerInstalls

Differences between revisions 9 and 59 (spanning 50 versions)
Revision 9 as of 2019-07-15 23:42:55
Size: 6988
Editor: mwhudson
Comment:
Revision 59 as of 2020-05-20 21:49:17
Size: 7094
Editor: mwhudson
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
||<#FF5555> '''This document is entirely a description of something that does not yet exist''' || = Automated Server Installs for 20.04 =
Line 3: Line 3:
= Automated Server Installs for 20.04 (and maybe 18.04.4?) =

||<#FF5555> '''This document is entirely a description of something that does not yet exist''' ||

Please direct feedback on this proposal to $TBD.
Please direct feedback on this new-in-20.04 feature to [[https://discourse.ubuntu.com/c/server|discourse.ubuntu.com]] or [[https://lists.ubuntu.com/mailman/listinfo/ubuntu-server|the ubuntu-server mailing list]].
Line 15: Line 11:
Autoinsallation lets you answer all those configuration questions ahead of time in an ''autoinstall file'' and lets the installation process run without any interaction. Autoinstallation lets you answer all those configuration questions ahead of time with ''autoinstall config'' and lets the installation process run without any interaction.
Line 23: Line 19:
 * the file format is completely different (yaml vs debconf-set-selections format)
 * when the answer to a question is not present in a preseed, d-i stops and asks the user for input. autoinstalls are not like this: if there is any autoinstall config at all, the installer takes the default for any unanswered question
 * the format is completely different (cloud-init config, usually yaml, vs debconf-set-selections format)
 * when the answer to a question is not present in a preseed, d-i stops and asks the user for input. autoinstalls are not like this: by default, if there is any autoinstall config at all, the installer takes the default for any unanswered question (and fails if there is no default). You can designate particular sections in the config as "interactive", which means the installer will still stop and ask about those.
Line 26: Line 22:
= Providing the autoinstall file = = Providing the autoinstall config =
Line 28: Line 24:
The autoinstall file can be provided in the following ways: The autoinstall config is provided via cloud-init configuration, which is almost endlessly flexible. In most scenarios the easiest way will be to provide user-data via the [[https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html|nocloud]] data source.
Line 30: Line 26:
 * As /autoinstall.cfg in the initrd
 * As /autoinstall.cfg on the install media (in any partition!)
 * Via a http or https (or maybe tftp) URL on the kernel command line
The autoinstall config should be provided under the 'autoinstall' key in the config. For example:
Line 34: Line 28:
= Creating an autoinstall file = {{{
#cloud-config
autoinstall:
    version: 1
    ...
}}}
Line 36: Line 35:
When any system is installed using the server installer, an autoinstall file for repeating the install is created at {{{/var/log/installer/autoinstall.cfg}}}. = Running a truly automatic autoinstall =
Line 38: Line 37:
Alternatively there is a snap, ``autoinstall-editor``, that can be used to either edit or create from scratch an autoinstall file (it is actually mostly the same code as that that runs the installation in interactive mode). Even if a fully noninteractive autoinstall config is found, the server installer will ask for confirmation before writing to the disks unless 'autoinstall' is present on the kernel command line. This is to make it harder to accidentally create a USB stick that will reformat a machine it is plugged into at boot. It is expected that most autoinstalls will be done via netboot, where the kernel command line is controlled by the netboot config -- just remember to put 'autoinstall' in there!

= Quick start =

So you just want to try it out? Well we have [[/QuickStart|the page for you]].

= Creating an autoinstall config =

When any system is installed using the server installer, an autoinstall file for repeating the install is created at {{{/var/log/installer/autoinstall-user-data}}}.

||<#FF5555> '''The snap described here does not yet exist''' ||

Alternatively there is a snap, ``autoinstall-editor``, that can be used to either edit or create from scratch an autoinstall config (it is actually mostly the same code as that that runs the installation in interactive mode).
Line 43: Line 54:
# dump out to stdout a complete autoinstall config file with default answers everywhere
$ autoinstall-editor --create
# edit existing autoinstall file
$ autoinstall-editor autoinstall.cfg
# dump out to stdout a complete autoinstall config with default answers everywhere
$ autoinstall-editor --create > my-autoinstall.yaml
# edit existing autoinstall config
$ autoinstall-editor my-autoinstall.yaml
Line 49: Line 60:
= The format of an autoinstall file = = The structure of an autoinstall config =
Line 51: Line 62:
The autoinstall file is YAML. Here is an example file that shows off most features: The autoinstall config has [[/ConfigReference|full documentation]].
Line 53: Line 64:
{{{#!yaml Technically speaking the config is not defined as a textual format, but cloud-init config is usually provided as YAML so that is the syntax the documentation uses.

A minimal config is:

{{{
Line 55: Line 70:
early_commands:
    - ping -c1 198.162.1.1
locale: en_US
keyboard:
    layout: en
    variant: uk
network:
    version: 2
    network:
        eth0:
            dhcp4: yes
proxy: http://squid.internal:3128/
mirror: http://repo.internal/
filesystem:
    recipe:
        name: lvm
Line 72: Line 71:
    username: mwhudson     hostname: hostname
    username: username
Line 74: Line 74:
ssh:
    authorized_keys:
      - $key
    allow_pw: no
snaps:
    - go/stable
debconf_selections: |
    bind9 bind9/run-resolvconf boolean false
packages:
    - libreoffice
    - dns-server^
late_commands:
    - touch /autoinstalled
Line 89: Line 76:
Many keys and values correspond straightforwardly to questions the installer asks (e.g. keyboard selection). There are some new options though: Here is an example file that shows off most features:
Line 91: Line 78:
 * `early_commands`: shell commands run after the installer has started, but before anything else (in particular, before scanning the system for block devices)
 * `debconf_selections` & `packages`: packages and configuration for them to be installed after installation has completed
 * `late_commands`: shell commands to run after the install has completed and any updates and packages installed, just before the system reboots
[[/ConfigReference#version|version]]`: 1`<<BR>>
[[/ConfigReference#reporting|reporting]]:<<BR>>
` hook:`<<BR>>
` type: webhook`<<BR>>
` endpoint: http://example.com/endpoint/path`<<BR>>
[[/ConfigReference#early-commands|early-commands]]`:`<<BR>>
` - ping -c1 198.162.1.1`<<BR>>
[[/ConfigReference#locale|locale]]`: en_US`<<BR>>
[[/ConfigReference#keyboard|keyboard]]`:`<<BR>>
` layout: en`<<BR>>
` variant: uk`<<BR>>
[[/ConfigReference#network|network]]`:`<<BR>>
` network:`<<BR>>
` version: 2`<<BR>>
` ethernets:`<<BR>>
` enp0s25:`<<BR>>
` dhcp4: yes`<<BR>>
` enp3s0: {}`<<BR>>
` enp4s0: {}`<<BR>>
` bonds:`<<BR>>
` bond0:`<<BR>>
` dhcp4: yes`<<BR>>
` interfaces:`<<BR>>
` - enp3s0`<<BR>>
` - enp4s0`<<BR>>
` parameters:`<<BR>>
` mode: active-backup`<<BR>>
` primary: enp3s0`<<BR>>
[[/ConfigReference#proxy|proxy]]`: http://squid.internal:3128/`<<BR>>
[[/ConfigReference#apt|apt]]`:`<<BR>>
` primary:`<<BR>>
` - arches: [default]`<<BR>>
` - uri: http://repo.internal/`<<BR>>
` sources:`<<BR>>
` my-ppa.list:`<<BR>>
` source: "deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu $RELEASE main"`<<BR>>
` keyid: B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77`<<BR>>
[[/ConfigReference#storage|storage]]`:`<<BR>>
` layout:`<<BR>>
` name: lvm`<<BR>>
[[/ConfigReference#identity|identity]]`:`<<BR>>
` hostname: hostname`<<BR>>
` username: username`<<BR>>
` password: $crypted_pass`<<BR>>
[[/ConfigReference#ssh|ssh]]`:`<<BR>>
` install-server: yes`<<BR>>
` authorized-keys:`<<BR>>
` - $key`<<BR>>
` allow-pw: no`<<BR>>
[[/ConfigReference#snaps|snaps]]`:`<<BR>>
` - name: go`<<BR>>
` channel: 1.14/stable`<<BR>>
` classic: true`<<BR>>
[[/ConfigReference#debconf-selections|debconf-selections]]`: |`<<BR>>
` bind9 bind9/run-resolvconf boolean false`<<BR>>
[[/ConfigReference#packages|packages]]`:`<<BR>>
` - libreoffice`<<BR>>
` - dns-server^`<<BR>>
[[/ConfigReference#user-data|user-data]]`:`<<BR>>
` disable_root: false`<<BR>>
[[/ConfigReference#late-commands|late-commands]]`:`<<BR>>
` - sed -ie 's/GRUB_TIMEOUT=.*/GRUB_TIMEOUT=30/' /target/etc/default/grub`<<BR>>
[[/ConfigReference#error-commands|error-commands]]`:`<<BR>>
` - tar c /var/log/installer | nc 192.168.0.1 1000`
Line 95: Line 143:
== Filesystem configuration == Many keys and values correspond straightforwardly to questions the installer asks (e.g. keyboard selection). See the reference for details of those that do not.
Line 97: Line 145:
Filesystem configuration is a complex topic and the description of the desired configuration in the autoinstall file can necessarily also be complex. The installer supports "layouts", simple ways of expressing common configurations. = Error handling =
Line 99: Line 147:
=== Supported layouts ===

The two supported layouts at the time of writing are "lvm" and "disk". Each supports a match spec (see below) to define which disk they apply to (if omitted, a disk is chosen arbitrarily).

{{{
filesystem:
  layout:
    name: lvm
    match:
      serial: CT*
filesystem:
  layout:
    name: disk
}}}

Supplying no filesystem config is equivalent to choosing the lvm layout with no disk match spec.

=== action-based config ===

For full flexibility, the installer allows filesystem configuration to be done using a syntax which is a superset of that supported by curtin, described at https://curtin.readthedocs.io/en/latest/topics/storage.html.

The extensions to the curtin syntax are mostly around disk selection. Curtin supported identifying disks by serial (e.g. `Crucial_CT512MX100SSD1_14250C57FECE`) or by path (e.g. `/dev/sdc`) and the server installer supports this as well. The installer additionally supports a ''match spec'' on a disk action that supports more flexible matching.

The actions in the filesystem config are processed in the order they are in the autoinstall file. Any disk action is assigned a matching disk -- chosen arbitrarily from the set of unassigned disks if there is more than one, and causing the installation to fail if there is no unassigned matching disk.

A match spec supports the following keys:

 * `model: foo`: matches a disk where ID_VENDOR=foo in udev, supporting globbing
 * `serial: foo`: matches a disk where ID_SERIAL=foo in udev, supporting globbing (the globbing support distinguishes this from specifying serial: foo directly in the disk action)
 * `ssd: yes|no`: matches a disk that is or is not an SSD (vs a rotating drive)
 * `size: largest`: take the largest disk rather than an arbitrary one if there are multiple matches
 * Anything else?

So for example, to match an arbitrary disk it is simply:

{{{
 - type: disk
   id: disk0
}}}

To match the largest ssd:

{{{
 - type: disk
   id: big-fast-disk
   match:
     ssd: yes
     size: largest
}}}

To match a Seagate drive:

{{{
 - type: disk
   id: data-disk
   match:
     model: Seagate
}}}

``autoinstall-editor`` supports creating and editing configs with arbitrary disk match specs.

Another extension to curtin syntax is how the size of partitions or LVM logical volumes are specified. Curtin just takes a size for this, but the server installer allows sizes to be specified as percentages of the containing device, or using min/priority/max as in d-i: (XXX I don't actually understand how min/priority/max works in d-i).

{{{
 - type: partition
   id: data-partition
   device: data-disk
   size: 75%
 - type: partition
   id: boot-partition
   device: boot-disk
   size:
     min: 512M
     priority: 600M
     max: 2G
}}}
Progress through the installer is reported via the [[ConfigReference#reporting|reporting]] system, including errors. In addition, when a fatal error occurs, the [[ConfigReference#error-commands]] are executed and the traceback printed to the console. The server then just waits.
Line 178: Line 151:
There are other places we could put the autoinstall config: We might want to extend the 'match specs' for disks to cover other ways of selecting disks.
Line 180: Line 153:
 * As a b64encoded gzipped blob on the kernel command line
 * Given as a URL via DHCP

Possibly the installer should support reporting progress to some endpoint.
We could support kickstart compatibility as in kickseed.

Automated Server Installs for 20.04

Please direct feedback on this new-in-20.04 feature to discourse.ubuntu.com or the ubuntu-server mailing list.

Introduction

The server installer for 20.04 supports a new mode of operation: automated installation, autoinstallation for short. You might also know this feature as unattended or handsoff or preseeded installation.

Autoinstallation lets you answer all those configuration questions ahead of time with autoinstall config and lets the installation process run without any interaction.

Differences from debian-installer preseeding

preseeds are the way to automate an installer based on debian-installer (aka d-i).

autoinstalls for the new server installer differ from preseeds in the following main ways:

  • the format is completely different (cloud-init config, usually yaml, vs debconf-set-selections format)
  • when the answer to a question is not present in a preseed, d-i stops and asks the user for input. autoinstalls are not like this: by default, if there is any autoinstall config at all, the installer takes the default for any unanswered question (and fails if there is no default). You can designate particular sections in the config as "interactive", which means the installer will still stop and ask about those.

Providing the autoinstall config

The autoinstall config is provided via cloud-init configuration, which is almost endlessly flexible. In most scenarios the easiest way will be to provide user-data via the nocloud data source.

The autoinstall config should be provided under the 'autoinstall' key in the config. For example:

#cloud-config
autoinstall:
    version: 1
    ...

Running a truly automatic autoinstall

Even if a fully noninteractive autoinstall config is found, the server installer will ask for confirmation before writing to the disks unless 'autoinstall' is present on the kernel command line. This is to make it harder to accidentally create a USB stick that will reformat a machine it is plugged into at boot. It is expected that most autoinstalls will be done via netboot, where the kernel command line is controlled by the netboot config -- just remember to put 'autoinstall' in there!

Quick start

So you just want to try it out? Well we have the page for you.

Creating an autoinstall config

When any system is installed using the server installer, an autoinstall file for repeating the install is created at /var/log/installer/autoinstall-user-data.

The snap described here does not yet exist

Alternatively there is a snap, autoinstall-editor, that can be used to either edit or create from scratch an autoinstall config (it is actually mostly the same code as that that runs the installation in interactive mode).

# start editing new config file
$ autoinstall-editor
# dump out to stdout a complete autoinstall config  with default answers everywhere
$ autoinstall-editor --create > my-autoinstall.yaml
# edit existing autoinstall config
$ autoinstall-editor my-autoinstall.yaml

The structure of an autoinstall config

The autoinstall config has full documentation.

Technically speaking the config is not defined as a textual format, but cloud-init config is usually provided as YAML so that is the syntax the documentation uses.

A minimal config is:

version: 1
identity:
    hostname: hostname
    username: username
    password: $crypted_pass

Here is an example file that shows off most features:

version: 1
reporting:
    hook:
        type: webhook
        endpoint: http://example.com/endpoint/path
early-commands:
    - ping -c1 198.162.1.1
locale: en_US
keyboard:
    layout: en
    variant: uk
network:
    network:
        version: 2
        ethernets:
            enp0s25:
               dhcp4: yes
            enp3s0: {}
            enp4s0: {}
        bonds:
            bond0:
                dhcp4: yes
                interfaces:
                    - enp3s0
                    - enp4s0
                parameters:
                    mode: active-backup
                    primary: enp3s0
proxy: http://squid.internal:3128/
apt:
    primary:
        - arches: [default]
        - uri: http://repo.internal/
    sources:
        my-ppa.list:
            source: "deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu $RELEASE main"
            keyid: B59D 5F15 97A5 04B7 E230  6DCA 0620 BBCF 0368 3F77
storage:
    layout:
        name: lvm
identity:
    hostname: hostname
    username: username
    password: $crypted_pass
ssh:
    install-server: yes
    authorized-keys:
      - $key
    allow-pw: no
snaps:
    - name: go
      channel: 1.14/stable
      classic: true
debconf-selections: |
    bind9      bind9/run-resolvconf    boolean false
packages:
    - libreoffice
    - dns-server^
user-data:
    disable_root: false
late-commands:
    - sed -ie 's/GRUB_TIMEOUT=.*/GRUB_TIMEOUT=30/' /target/etc/default/grub
error-commands:
    - tar c /var/log/installer | nc 192.168.0.1 1000

Many keys and values correspond straightforwardly to questions the installer asks (e.g. keyboard selection). See the reference for details of those that do not.

Error handling

Progress through the installer is reported via the reporting system, including errors. In addition, when a fatal error occurs, the ConfigReference#error-commands are executed and the traceback printed to the console. The server then just waits.

Possible future directions

We might want to extend the 'match specs' for disks to cover other ways of selecting disks.

We could support kickstart compatibility as in kickseed.

FoundationsTeam/AutomatedServerInstalls (last edited 2020-06-11 04:19:11 by mwhudson)