Encrypted ZFS Root on Ubuntu Server 20.04 (with USB Unlock)

It's all nice and dandy to setup unencrypted ZFS on server or setting it up with boot encryption. However, what if we want to use USB to unlock the encrypted drive? And no, it's not as crazy as it seems. Scenarios are actually plentiful.

One scenario is when you have your servers encrypted (as realistically everybody should) but you don't necessarily want (or can) enter the password. If you can plug in USB with a key and make ZFS use that key, you suddenly have password-less boot without connecting to the server. After boot is done, you can unplug the USB and store it somewhere safe. And this can be done by literally anybody you trust - it doesn't have to be you.

My favorite scenario is using it with self-erasing USB drive. You place the encryption key on the small drive and it will be there for every boot. You can use your server as you normally would. However, if power is lost, your key will disappear and content of server will not be accessible anymore. When would such crazy scenario happen you ask? Well, if anybody is stealing your server they have to unplug it first. And yes, your server is gone but at least your data is not.

I admit I never had that scenario happen to me - fortunately all my servers are still accounted for. But I did RMA disk drives. And worrying about erasing the data when you cannot access it anymore is a bit too late.

Whatever might be your case, let me guide you through setting up natively-encrypted ZFS with a key on the USB drive.

Once you enter the shell of the installation media, the very first step is setting up a few variables - location of disk and USB drive, followed by pool and host name. This way we can use them going forward and avoid accidental mistakes. Make sure to replace these values with the ones appropriate for your system.

Terminal
DISK=/dev/disk/by-id/ata-xxx
USB=/dev/disk/by-id/usb-xxx
POOL=Ubuntu
HOST=server

Next let's sort out question of the encryption key. Assumption is that the key will be on the first partition of the FAT formatted USB drive and we'll mount it at /tmpusb. While you could create the key material directly, I personally prefer the passphrase as it makes life easier in the case of recovery. If you already have the passphrase on the drive just skip the last command as it will overwrite it.

Terminal
mkdir /tmpusb
mount -t vfat -o rw "$USB-part1" /tmpusb
echo -n "password" > /tmpusb/boot.pwd

General idea of my disk setup is to maximize amount of space available for pool with the minimum of supporting partitions. If you are installing on SSD blkdiscard will trim all the data. You can safely ignore any errors on disks that don't support it.

Terminal
blkdiscard $DISK 2>/dev/null

sgdisk --zap-all $DISK

sgdisk -n1:1M:+127M -t1:EF00 -c1:EFI $DISK
sgdisk -n2:0:+512M -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:0 -t3:8309 -c3:Ubuntu $DISK

sgdisk --print $DISK

To kick off the fun of the installation we need debootstrap and zfsutils-linux package.

Terminal
apt update
apt install --yes debootstrap zfsutils-linux

Now we're ready to create system ZFS pool.

Terminal
zpool create -o ashift=12 -o autotrim=on \
-O compression=lz4 -O normalization=formD \
-O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
-O encryption=aes-256-gcm -O keyformat=passphrase -O keylocation=file:///tmpusb/boot.pwd \
-O canmount=off -O mountpoint=none -R /mnt/install $POOL $DISK-part3
zfs create -o canmount=noauto -o mountpoint=/ $POOL/System
zfs mount $POOL/System

Assuming UEFI boot, two additional partitions are needed - one for EFI and one for booting. I don't have ZFS pool for boot partition but a plain old ext4 as I find potential fixup works better that way.

Terminal
yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

Bootstrapping Ubuntu on the newly created pool is next. This will take a while.

Terminal
debootstrap focal /mnt/install/

zfs set devices=off $POOL

Our newly copied system is lacking a few files and we should make sure they exist before proceeding.

Terminal
echo $HOST > /mnt/install/etc/hostname
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

Finally we're ready to "chroot" into our new system.

Terminal
mount --rbind /dev /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys /mnt/install/sys
chroot /mnt/install /usr/bin/env DISK=$DISK USB=$USB POOL=$POOL bash --login

Let's not forget to setup locale and time zone.

Terminal
locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

To mount EFI and boot partitions, we need to do some fstab setup too:

Terminal
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
/boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
/boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we're ready to onboard the latest Linux image.

Terminal
apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by the boot environment packages.

Terminal
apt install --yes zfs-initramfs plymouth grub-efi-amd64-signed shim-signed

Now it's time to setup boot scripts to ensure USB drive is mounted before ZFS needs it. I found that initramfs' init-premount directory is the ideal spot.

Terminal
cat << EOF > /usr/share/initramfs-tools/scripts/init-premount/tmpusb
#!/bin/sh -e

PREREQ="udev"
prereqs() {
echo "\$PREREQ"
}

case \$1 in
prereqs)
prereqs
exit 0
;;
esac

USB="$USB"
POOL="$POOL"

echo "Waiting for \$USB"
for I in \`seq 1 20\`; do
if [ -e "\$USB" ]; then break; fi
echo -n .
sleep 1
done
echo

sleep 2

if [ -e "\$USB" ]; then
mkdir /tmpusb
mount -t vfat -o ro "\$USB-part1" /tmpusb
if [ \$? -eq 0 ]; then
exit 0
else
echo "Error mounting \$USB-part1" >&2
fi
else
echo "Cannot find \$USB" >&2
fi
exit 1
EOF
chmod 755 /usr/share/initramfs-tools/scripts/init-premount/tmpusb

cat << EOF > /usr/share/initramfs-tools/scripts/init-bottom/tmpusb
#!/bin/sh -e

PREREQ="udev"
prereqs() {
echo "\$PREREQ"
}

case \$1 in
prereqs)
prereqs
exit 0
;;
esac

if [ -e "/tmpusb" ]; then
umount /tmpusb
rmdir /tmpusb
fi
EOF
chmod 755 /usr/share/initramfs-tools/scripts/init-bottom/tmpusb

The first script will wait for USB drive if needed and mount it at /tmpusb for ZFS to find. Second script is there just for a bit of cleanup.

If USB drive is not mounted, this will cause boot to fail. If we want ZFS to ask for the passphrase instead (despite having file as the keylocation) a further customization is needed. But note these commands might need adjustment and they definitely need to be repeated each time ZFS package is updated. I might go into the details in some future post but suffice to say this is really not future-proof solution but it's the minimum set of changes that I could make sed work with.

Terminal
sed -i 's/load-key/load-key -L prompt/' /usr/share/initramfs-tools/scripts/zfs
sed -i '0,/load-key/ {s/-L prompt//}' /usr/share/initramfs-tools/scripts/zfs
sed -i '/KEYSTATUS=/i \\t\t\t$ZFS load-key "${ENCRYPTIONROOT}"' /usr/share/initramfs-tools/scripts/zfs
sed -i '/KEYSTATUS=/i \\t\t\tKEYLOCATION=prompt' /usr/share/initramfs-tools/scripts/zfs

In lieu of warning, suffice it to say these changes to zfs script are suitable only for this scenario and don't really work for anything else.

Now we get grub started and update our boot environment. Due to Ubuntu 19.10 having some kernel version kerfuffle, we need to manually create initramfs image. This is also a good moment to check if our script is in.

Terminal
KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

lsinitramfs /boot/initrd.img-$KERNEL | grep tmpusb

Grub update is what makes EFI tick.

Terminal
update-grub 2>/dev/null
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
--recheck --no-floppy

Short package upgrade will not hurt.

Terminal
apt dist-upgrade --yes

We can omit creation of the swap dataset but I personally find its good to have it just in case.

Terminal
zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput \
-o sync=always -o primarycache=metadata -o secondarycache=none $POOL/Swap
mkswap -f /dev/zvol/$POOL/Swap
echo "/dev/zvol/$POOL/Swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

This is a good time to install other packages (e.g. openssh-server) and do any setup you might need (e.g. firewall). If nothing else, then setup root password so you have a way to log in (I personally prefer to create another user and leave root passwordless).

Terminal
passwd

As installation is finally done, we can exit our chroot environment.

Terminal
exit

And cleanup mount points.

Terminal
umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

After the reboot you should be able to enjoy your installation.

Terminal
reboot

3 thoughts to “Encrypted ZFS Root on Ubuntu Server 20.04 (with USB Unlock)”

    1. By adding another device, are you adding it to existing pool or you are talking about the new pool? If it’s the same pool, I have suspicion that your boot/initramfs data is not synced between disks and system simply tries to boot from the wrong one. If so, dd-ing efi/boot partition will do the trick.

  1. Hi,
    Those scripts for a USB drive mount do not work on Ubuntu 22.04.2. There are errors during booting: usb 1-2.3: device descriptor read/B, error -110
    The system is already installed on ZFS, I just try to change the keyformat passphrase to an USB key. Looking for any solutions.

Leave a Reply

Your email address will not be published. Required fields are marked *