Installing Ubuntu 16.04 on a ZFS root filesystem

One of the major new pieces of functionality in Ubuntu 16.04 (Xenial) is built in support for ZFS filesystems.  I was disappointed to learn that ZFS support is not actually built into the installer itself, leaving you to piece things together yourself.  I did find a few good tutorials online for this, but they all seem to be missing a few pieces.  I'm hoping that this guide will be a bit more complete for people.

It is based on a combination of the following two guides:

• https://github.com/zfsonlinux/pkg-zfs/wiki/HOWTO-install-Ubuntu-16.04-to-a-Native-ZFS-Root-Filesystem
• http://dotfiles.tnetconsulting.net/articles/2016/0327/ubuntu-zfs-native-root.html

Booting Into the Install Environment

Setting up a ZFS filesystem requires a full set of userspace tools rather than the limited set included within the actual installer.  Due to this, we're going to boot a Ubuntu Desktop live CD, and do a manual installation within its root filesystem.  I found that the easiest way to do this was to boot the live CD, set a password for the ubuntu user, and SSH into the machine from a remote box.  This way it is much easier to copy/paste commands from the webpage.

To start off, we need to install the ZFS tools and debootstrap which we will use for actually installing the operating system:

apt-add-repository universe
apt-get update
apt-get install --yes zfsutils-linux debootstrap

Partitioning the Disks

Next, we partition the disks.  For now we're going to assume a two disk system which is just doing striping, but you can change as desired:

parted -- /dev/sda mklabel msdos Y mkpart primary zfs 0% 100%
parted -- /dev/sdb mklabel msdos Y mkpart primary zfs 0% 100%

Device naming during bootup on Linux isn't always static, so we want to reference the disks through a static identifier like disk ID (/dev/disk/by-id), unfortunately Grub fails to properly identify the devices from the zfs command output, so we need this hack for now to make sure that update-grub will correctly identify the disks.  Specifically, it forces creation of symlinks in /dev which map to the device names in /dev/disk/by-id.  Some additional details about the issue can be found in this grub2 launchpad bug.

echo 'KERNEL=="sd*[!0-9]", IMPORT{parent}=="ID_*", SYMLINK+="$env{ID_BUS}-$env{ID_SERIAL}"
> KERNEL=="sd*[0-9]", IMPORT{parent}=="ID_*", SYMLINK+="$env{ID_BUS}-$env{ID_SERIAL}-part%n"' > /etc/udev/rules.d/90-zfs.rules
udevadm trigger

Create Your Zpool

Now we actually create the zpool and import it to /mnt:

zpool create -m none -o ashift=12 -O compression=lz4 rpool /dev/sda1 /dev/sdb1
zfs create -o mountpoint=/ rpool/root
zpool export rpool
zpool import -d /dev/disk/by-id -R /mnt rpool

So here we create the zpool by device name, and then re-import it by device ID while mounting at /mnt.

And then create various partitions off of the root filesystem:

zfs create -o setuid=off rpool/root/home
zfs create -o mountpoint=/root rpool/root/home/root
zfs create -o canmount=off -o setuid=off -o exec=off rpool/root/var
zfs create -o com.sun:auto-snapshot=false rpool/root/var/cache
zfs create rpool/root/var/log
zfs create rpool/root/var/spool
zfs create -o com.sun:auto-snapshot=false -o exec=on rpool/root/var/tmp

We break out these various subdirectories so we have the ability to optionally enable/disable compression, support for setuid/exec, and various other options.  We can also choose to limit the maximum size of each mount independently.

Install Ubuntu

The install of Ubuntu itself is pretty straightforward:

debootstrap the OS

debootstrap xenial /mnt

zfs set devices=off rpool

grep -v cdrom /etc/apt/sources.list > /mnt/etc/apt/sources.list
cp /etc/udev/rules.d/90-zfs.rules /mnt/etc/udev/rules.d/90-zfs.rules

You'll notice at the end here we copy our hacky udev rule to the new filesystem.  The "devices=off" option we set here disables the ability to create device nodes in the filesystem.  This works since /dev is actually a devtmpfs partition of its own.

Configure the network interface

export INTERFACE=$(ip addr list | grep ^[0-9]: | grep -v "lo" | awk {'print $2'} | cut -d ":" -f 1)

echo test > /mnt/etc/hostname

echo 127.0.1.1  >> /mnt/etc/hosts

echo "auto $INTERFACE

iface $INTERFACE inet dhcp" > /mnt/etc/network/interfaces.d/$INTERFACE

Enter the chroot for some final setup

mount --rbind /dev /mnt/dev

mount --rbind /proc /mnt/proc

mount --rbind /sys /mnt/sys

chroot /mnt /bin/bash --login

locale-gen en_US.UTF-8

echo 'LANG="en_US.UTF-8"' > /etc/default/locale

apt-get update

apt-get install --yes zfsutils-linux zfs-initramfs grub-pc linux-image-generic ssh

dpkg-reconfigure tzdata

update-initramfs -c -k all

Set Up Grub's configuration

sed -i 's/^\(GRUB_CMDLINE_LINUX_DEFAULT\)=.*/\1=""/g' /etc/default/grub

sed -i 's|^\(GRUB_HIDDEN_TIMEOUT=.*\)|#\1|g' /etc/default/grub

sed -i 's/^\(GRUB_CMDLINE_LINUX\)="\(.*\)"/\1="boot=zfs \2"/g' /etc/default/grub

ln -s /proc/mounts  /etc/mtab

update-grub

rm /etc/mtab

Set a root password and exit the chroot

passwd root

exit

Install the grub bootloader

grub-probe /mnt

grub-install --root-directory=/mnt /dev/sda

grub-install --root-directory=/mnt /dev/sdb

Reboot into the environment

reboot