Medo's Home Page

No. That’s the short of it.

Longer version starts with me setting up my new Ubuntu server and deciding to use netplan to setup network interfaces. Nothing too big. Just a few bonds here and there. And all seemed fine until I tried to setup email and noticed the following message in syslog:

postfix/smtp[49721]: connect to alt1.aspmx.l.google.com[209.85.146.27]:25: Connection timed out
postfix/smtp[49721]: connect to aspmx5.googlemail.com[2607:f8b0:4002:c08::1b]:25: Network is unreachable

Once I double-checked, I noticed syslog was right – I only had link-local address assigned to interface. Ok, so DHCPv6 only turns on DHCP and not prefix delegation I use for my network. Easy-peasy, that’s surely matter of just turning on the correct setting…

Or finding a matching bug. Yep, prefix delegation that’s perfectly valid and well supported IPv6 address allocation method is not supported.

But ticket did solve my problem. As advised, I simply reverted to use networkd (don’t forget to use systemctl enable systemd-networkd).

PS: Here are my networkd files that work with IPv6 PD.

/etc/systemd/network/10-bond0.netdev 
[NetDev]
Name=bond0
MACAddress=00:25:90:ba:31:40
Kind=bond

[Bond]
Mode=802.3ad
LACPTransmitRate=fast
MIIMonitorSec=100ms
TransmitHashPolicy=layer2+3

/etc/systemd/network/10-bond0.network 
[Match]
Name=bond0

[DHCP]
RouteMetric=100
UseMTU=true

[Network]
DHCP=yes
IPv6PrefixDelegation=yes
IPv6PrivacyExtensions=true
LinkLocalAddressing=ipv6
ConfigureWithoutCarrier=yes

/etc/systemd/network/11-eno1.network 
[Match]
PermanentMACAddress=00:25:90:ba:31:40

[Network]
Bond=lan

/etc/systemd/network/12-eno2.network 
[Match]
PermanentMACAddress=00:25:90:ba:31:41

[Network]
Bond=lan

It’s all nice and dandy to setup unencrypted ZFS on server or setting it up with boot encryption. However, what if we want to use USB to unlock the encrypted drive? And no, it’s not as crazy as it seems. Scenarios are actually plentiful.

One scenario is when you have your servers encrypted (as realistically everybody should) but you don’t necessarily want (or can) enter the password. If you can plug in USB with a key and make ZFS use that key, you suddenly have password-less boot without connecting to the server. After boot is done, you can unplug the USB and store it somewhere safe. And this can be done by literally anybody you trust – it doesn’t have to be you.

My favorite scenario is using it with self-erasing USB drive. You place the encryption key on the small drive and it will be there for every boot. You can use your server as you normally would. However, if power is lost, your key will disappear and content of server will not be accessible anymore. When would such crazy scenario happen you ask? Well, if anybody is stealing your server they have to unplug it first. And yes, your server is gone but at least your data is not.

I admit I never had that scenario happen to me – fortunately all my servers are still accounted for. But I did RMA disk drives. And worrying about erasing the data when you cannot access it anymore is a bit too late.

Whatever might be your case, let me guide you through setting up natively-encrypted ZFS with a key on the USB drive.

Once you enter the shell of the installation media, the very first step is setting up a few variables – location of disk and USB drive, followed by pool and host name. This way we can use them going forward and avoid accidental mistakes. Make sure to replace these values with the ones appropriate for your system.

Terminal 
DISK=/dev/disk/by-id/ata-xxx
USB=/dev/disk/by-id/usb-xxx
POOL=Ubuntu
HOST=server

Next let’s sort out question of the encryption key. Assumption is that the key will be on the first partition of the FAT formatted USB drive and we’ll mount it at /tmpusb. While you could create the key material directly, I personally prefer the passphrase as it makes life easier in the case of recovery. If you already have the passphrase on the drive just skip the last command as it will overwrite it.

Terminal 
mkdir /tmpusb
mount -t vfat -o rw "$USB-part1" /tmpusb
echo -n "password" > /tmpusb/boot.pwd

General idea of my disk setup is to maximize amount of space available for pool with the minimum of supporting partitions. If you are installing on SSD blkdiscard will trim all the data. You can safely ignore any errors on disks that don’t support it.

Terminal 
blkdiscard $DISK 2>/dev/null

sgdisk --zap-all                        $DISK

sgdisk -n1:1M:+127M -t1:EF00 -c1:EFI    $DISK
sgdisk -n2:0:+512M  -t2:8300 -c2:Boot   $DISK
sgdisk -n3:0:0      -t3:8309 -c3:Ubuntu $DISK

sgdisk --print                          $DISK

To kick off the fun of the installation we need debootstrap and zfsutils-linux package.

Terminal 
apt install --yes debootstrap zfsutils-linux

Now we’re ready to create system ZFS pool.

Terminal 
zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O encryption=aes-256-gcm -O keyformat=passphrase -O keylocation=file:///tmpusb/boot.pwd \
    -O canmount=off -O mountpoint=none -R /mnt/install $POOL $DISK-part3
zfs create -o canmount=noauto -o mountpoint=/ $POOL/System
zfs mount $POOL/System

Assuming UEFI boot, two additional partitions are needed – one for EFI and one for booting. I don’t have ZFS pool for boot partition but a plain old ext4 as I find potential fixup works better that way.

Terminal 
yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

Bootstrapping Ubuntu on the newly created pool is next. This will take a while.

Terminal 
debootstrap focal /mnt/install/

zfs set devices=off $POOL

Our newly copied system is lacking a few files and we should make sure they exist before proceeding.

Terminal 
echo $HOST > /mnt/install/etc/hostname
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

Finally we’re ready to “chroot” into our new system.

Terminal 
mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install /usr/bin/env DISK=$DISK USB=$USB POOL=$POOL bash --login

Let’s not forget to setup locale and time zone.

Terminal 
locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

To mount EFI and boot partitions, we need to do some fstab setup too:

Terminal 
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we’re ready to onboard the latest Linux image.

Terminal 
apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by the boot environment packages.

Terminal 
apt install --yes zfs-initramfs plymouth grub-efi-amd64-signed shim-signed

Now it’s time to setup boot scripts to ensure USB drive is mounted before ZFS needs it. I found that initramfs’ init-premount directory is the ideal spot.

Terminal 
cat << EOF > /usr/share/initramfs-tools/scripts/init-premount/tmpusb
#!/bin/sh -e

PREREQ="udev"
prereqs() {
    echo "\$PREREQ"
}

case \$1 in
    prereqs)
        prereqs
        exit 0
    ;;
esac

USB="$USB"
POOL="$POOL"

echo "Waiting for \$USB"
for I in \`seq 1 20\`; do
    if [ -e "\$USB" ]; then break; fi
    echo -n .
    sleep 1
done
echo

sleep 2

if [ -e "\$USB" ]; then
    mkdir /tmpusb
    mount -t vfat -o ro "\$USB-part1" /tmpusb
    if [ \$? -eq 0 ]; then
        exit 0
    else
        echo "Error mounting \$USB-part1" >&2
    fi
else
    echo "Cannot find \$USB" >&2
fi
exit 1
EOF
chmod 755 /usr/share/initramfs-tools/scripts/init-premount/tmpusb

cat << EOF > /usr/share/initramfs-tools/scripts/init-bottom/tmpusb
#!/bin/sh -e

PREREQ="udev"
prereqs() {
    echo "\$PREREQ"
}

case \$1 in
    prereqs)
        prereqs
        exit 0
    ;;
esac

if [ -e "/tmpusb" ]; then
    umount /tmpusb
    rmdir /tmpusb
fi
EOF
chmod 755 /usr/share/initramfs-tools/scripts/init-bottom/tmpusb

The first script will wait for USB drive if needed and mount it at /tmpusb for ZFS to find. Second script is there just for a bit of cleanup.

If USB drive is not mounted, this will cause boot to fail. If we want ZFS to ask for the passphrase instead (despite having file as the keylocation) a further customization is needed. But note these commands might need adjustment and they definitely need to be repeated each time ZFS package is updated. I might go into the details in some future post but suffice to say this is really not future-proof solution but it’s the minimum set of changes that I could make sed work with.

Terminal 
sed -i 's/load-key/load-key -L prompt/' /usr/share/initramfs-tools/scripts/zfs
sed -i '0,/load-key/ {s/-L prompt//}' /usr/share/initramfs-tools/scripts/zfs
sed -i '/KEYSTATUS=/i \\t\t\t$ZFS load-key "${ENCRYPTIONROOT}"' /usr/share/initramfs-tools/scripts/zfs
sed -i '/KEYSTATUS=/i \\t\t\tKEYLOCATION=prompt' /usr/share/initramfs-tools/scripts/zfs

In lieu of warning, suffice it to say these changes to zfs script are suitable only for this scenario and don’t really work for anything else.

Now we get grub started and update our boot environment. Due to Ubuntu 19.10 having some kernel version kerfuffle, we need to manually create initramfs image. This is also a good moment to check if our script is in.

Terminal 
KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

lsinitramfs /boot/initrd.img-$KERNEL | grep tmpusb

Grub update is what makes EFI tick.

Terminal 
update-grub 2>/dev/null
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Short package upgrade will not hurt.

Terminal 
apt dist-upgrade --yes

We can omit creation of the swap dataset but I personally find its good to have it just in case.

Terminal 
zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput \
    -o sync=always -o primarycache=metadata -o secondarycache=none $POOL/Swap
mkswap -f /dev/zvol/$POOL/Swap
echo "/dev/zvol/$POOL/Swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

This is a good time to install other packages (e.g. openssh-server) and do any setup you might need (e.g. firewall). If nothing else, then setup root password so you have a way to log in (I personally prefer to create another user and leave root passwordless).

Terminal 
passwd

As installation is finally done, we can exit our chroot environment.

Terminal 
exit

And cleanup mount points.

Terminal 
umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

After the reboot you should be able to enjoy your installation.

Terminal 
reboot

Dark Mode for QT Application

Getting Rid of Aspeed Firmware Warning

IPv6 Prefix Delegation with NetPlan

Encrypted ZFS Root on Ubuntu Server 20.04 via USB Unlock