Overview

This documents how to set up a Xen virtualization host to be usable via libvirt from virt-manager. This also allows manual commands issued on the host but does not cover integration/usage via XCP or OpenStack. The descriptions are taken from a 12.04 (Precise) install, though most should be also/still valid for Oneiric and Quantal.

To manage the Xen host via virt-manager/libvirt has the advantage that the virtualization host does not need a GUI. So usually the host is a ubuntu-server based installation. It might be possible to have a desktop install and add the Xen hypervisor and virt-manager on the same machine (I have not tested that and I am not sure how well the desktop runs within dom0 as it can differ from bare metal).

+-----------------------+             +--------------------------+
|      Mgmnt. Host      |             |         Xen Host         |
|                       |             |                          |
|+-----+ +-----+ +-----+|             |+----+ +----+ +----+ +---+|
|| VNX | | VNX | | ... ||             ||dom0| |domU| |domU| |...||
|+-----+ +-----+ +-----+|             |+-+--+ +-+--+ +-+--+ +-+-+|
|   |       |       |   |             |  |      |      |      |  |
|+---------------------+|             |+-+------+------+------+-+|
||    virt-manager     ||             ||         libvirt        ||
|+----------+----------+|             |+------------+-----------+|
+-----------|-----------+             +-------------|------------+
            +---------------------------------------+
                              SSH

Xen Host Setup

Installation

•   sudo apt-get install openssh-server^ virt-host^ xen-system-<arch>

Older releases may not provide the xen-system-<arch> meta-package (use xen-hypervisor-<arch> instead) and starting with 13.10 (Saucy) there is no i386 hypervisor. So Xen can only be used on a 64bit capable cpu. The base installation can be 32bit, though. The xen-system-amd64 meta-package has to be used for both.

This installs an ssh server (for communication), the libvirt libraries and tools (including bridge-utils) and the Xen hypervisor (<arch> is either i386 or amd64). By default libvirt is set up to provide one virtual NAT-bridged network (virbr0) which can be used if guests are not required to be accessible from outside (a DHCP server inside that virtual network is configured as well).

Configuration

Grub Setup

Grub can be changed to automatically boot into Xen mode (which is a automatically added as a sub-menu to grub) and the grub config srcipts understand a few additional options in /etc/default/grub:

•   # GRUB_DEFAULT can be a string
    GRUB_DEFAULT="Xen 4.1-<arch>"
  
    # Replacements for the kernel paramenters GRUB_CMDLINE_LINUX[_DEFAULT].
    # Those get passed to the dom0 kernel.
    GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT="console=hvc0 console=tty0"
    GRUB_CMDLINE_LINUX_XEN_REPLACE=""
  
    # Those get passed to the Xen mini-os
    GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=512M,max:512M com1=115200,8n1 console=com1,vga"
    GRUB_CMDLINE_XEN=""

The string required as GRUB_DEFAULT changed between releases. It can be found in /boot/grub/grub.cfg.

This allows to have certain kernel arguments when booting a kernel bare-metal and a different set of options for the dom0 kernel. This is quite useful when trying to re-direct console output. A full list of boot options can be found in [ 1 ].

The console statements only make sense together. The Xen arguments com1 and console create and use the hypervisor console which the dom0 kernel argument will tell the dom0 kernel to use as well. Of course the host needs some serial port for that.

The example dom0_mem option will limit the memory for dom0 to 512MB of memory as described in [ 2 ]. This prevents memory ballooning.

Memory in Xen can not be overcommitted.

Running sudo update-grub will create a new grub configuration with those settings.

Xen Configuration

There are two toolstacks that come with xen:

• xm: which is deprecated but right now it is still needed (see below)
• xl: which is the successor to xm

The default is xm but can be changed in /etc/default/xen (the system needs a restart after this).

Starting with 13.10 (Saucy) libvirt works with both stacks. Previous releases only work with the xm toolstack.

XM Toolstack

This stack interacts with Xend (a daemon written in Python). Its settings can be changed in /etc/xen/xend-config.sxp:

•  ...
   # Disable ballooning
   (dom0-min-mem 512)
   (enable-dom0-ballooning no)
   ...
   # This is necessary to allow libvirt to access xend
   (xend-unix-server yes)
   ...
   # Specify the name of the default bridge.
   (vif-script vif-bridge bridge=virbr0)
   ...

The amount of memory for dom0-min-memory should be the same as specified by the dom0 command line option. Disabling any dom0 ballooning is described in [ 2 ].

Older Xen installations would recommend to allow its scripts to set up bridges. But now [ 2 ] strongly suggests to let the OS create any bridges. Naming the default bridge here avoids the need of making that setting in manually created guest config files.

Transparent Bridging

When Xen guests should integrate into an existing local network (for example to make use of common PXE infrastructure, or to appear as normal local machines), a transparent bridge is required. This can easily be created (bridge-utils was already installed by the virt-host task). Changes go into /etc/network/interfaces:

•  auto eth0
   iface eth0 inet manual
  
   auto br0
   iface br0 inet static
          address         192.168.1.2
          netmask         255.255.255.0
          broadcast       192.168.1.255
          gateway         192.168.1.1
          dns-nameservers 192.168.1.1
          dns-search      <localdomain>
          bridge-ports    eth0
          bridge-stp      off
          bridge-fd       0
          bridge-maxwait  0
          post-up         ip link set br0 address xx:xx:xx:xx:xx:xx

Setting the MAC address of the bridge manually apparently avoids it from changing to be always the lowest address of connected ports. A safe value is the address of eth0.

Declaring DNS nameservers here and per interface became important with changes to resolvconf in Precise.

User Setup

Libvirt can allow normal users to manage virtual machines, storage pools and virtual networks on a virtualization host. For that the user needs an account on the host and needs to be added to the libvirtd system group.

In order to verify whether setup was successful so far, try the following on the Xen host:

•  #> virsh -c xen:/// dominfo 0
   Id:             0
   Name:           Domain-0
   UUID:           00000000-0000-0000-0000-000000000000
   OS Type:        linux
   State:          running
   CPU(s):         ...

Management Host Setup

Installation

•  sudo apt-get install virt-manager

Configuration

After starting virt-manager, a new connection has to be added via File->Add Connection.... Hypervisor is Xen, "Connect to remote host" selected, Method is SSH and User- and Hostname set to whatever was configured.

When the connection is successful, its name and (Xen) will appear in the main window along with dom0 showing up as running. By right-clicking on the hostname and selecting Details from the context window the connection details are shown. From there it is possible to declare additional storage pools and virtual networks.

Xen does not access disk image files directly. It does automatically create loop devices for image files and uses those. Thus using logical volumes for disk images is a bit less indirect. Having a volume group defined on the Xen host with spare extends and declaring that as a pool from virt-manager does work quite well (at least since Oneiric).

References

1. Xen Hypervisor Boot Options

2. Xen Best Practices