Booting Encrypted ZFS Mirror on Ubuntu 18.10

As I was setting up my new Linux machine with two disks, I decided to forgo my favorite Linux Mint and give Ubuntu another try. Main reason? ZFS of course.

Ubuntu already has a quite decent guide for ZFS setup but it’s slightly lacking in the mirroring department. So, here I will list steps that follow their approach closely but with slight adjustments as not only I want encrypted setup but also a proper ZFS mirror setup. If you need a single disk ZFS setup, stick with the original guide.

After booting into installation, we can go for Try Ubuntu and open a terminal. My strong suggestion would be to install openssh-server package first and connect to it remotely because that allows for copy/paste:

Terminal
passwd
Changing password for ubuntu.
(current) UNIX password: (empty)
Enter new UNIX password: password
Retype new UNIX password: password
passwd: password updated successfully

sudo apt install --yes openssh-server

Regardless if you continue directly or you connect via SSH (username is ubuntu), the first task is to get onto root prompt and never leave it again. :)

Terminal
sudo -i

To get the ZFS on, we need Internet connection and extra repository:

Terminal
sudo apt-add-repository universe
apt update

Now we can finally install ZFS, partitioning utility, and an installation tool:

Terminal
apt install --yes debootstrap gdisk zfs-initramfs

First we clean the partition table on disks followed by a few partition definitions (do change ID to match your disks):

Terminal
sgdisk --zap-all /dev/disk/by-id/ata_disk1
sgdisk --zap-all /dev/disk/by-id/ata_disk2

sgdisk -a1 -n2:34:2047 -t2:EF02 /dev/disk/by-id/ata_disk1
sgdisk -a1 -n2:34:2047 -t2:EF02 /dev/disk/by-id/ata_disk2

sgdisk -n3:1M:+512M -t3:EF00 /dev/disk/by-id/ata_disk1
sgdisk -n3:1M:+512M -t3:EF00 /dev/disk/by-id/ata_disk2

sgdisk -n4:0:+512M -t4:8300 /dev/disk/by-id/ata_disk1
sgdisk -n4:0:+512M -t4:8300 /dev/disk/by-id/ata_disk2

sgdisk -n1:0:0 -t1:8300 /dev/disk/by-id/ata_disk1
sgdisk -n1:0:0 -t1:8300 /dev/disk/by-id/ata_disk2

After all these we should end up with both disks showing 4 distinct partitions:

Terminal
sgdisk --print /dev/disk/by-id/ata_disk1

Number Start (sector) End (sector) Size Code Name
1 2099200 67108830 31.0 GiB 8300
2 34 2047 1007.0 KiB EF02
3 2048 1050623 512.0 MiB EF00
4 1050624 2099199 512.0 MiB 8300

With partitioning done, it’s time to encrypt our disks and mount them (note that we only encrypt the first partition, not the whole disk):

Terminal
cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 /dev/disk/by-id/ata_disk1-part1
cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 /dev/disk/by-id/ata_disk2-part1

cryptsetup luksOpen /dev/disk/by-id/ata_disk1-part1 luks1
cryptsetup luksOpen /dev/disk/by-id/ata_disk2-part1 luks2

Finally we can create our pool (rpool is a “standard” name) consisting of both encrypted devices:

Terminal
zpool create -o ashift=12 -O atime=off -O compression=lz4 \
-O normalization=formD -O xattr=sa -O mountpoint=/ -R /mnt/rpool \
rpool mirror /dev/mapper/luks1 /dev/mapper/luks2

There is advantage into creating fine grained datasets as the official guide instructs, but I personally don’t do it. Having one big free-for-all pile is OK for me – anything of any significance I anyhow keep on my network drive where I have properly setup ZFS with rights, quotas, and all other goodies.

Since we are using LUKS encryption, we do need to mount 4th partition too. We’ll do it for both disks and deal with syncing them later:

Terminal
mkdir /mnt/rpool/boot
mke2fs -t ext2 /dev/disk/by-id/ata_disk1-part4
mount /dev/disk/by-id/ata_disk1-part4 /mnt/rpool/boot

mkdir /mnt/rpool/boot2
mke2fs -t ext2 /dev/disk/by-id/ata_disk2-part4
mount /dev/disk/by-id/ata_disk2-part4 /mnt/rpool/boot2

Now we can finally start copying our Linux (do check for current release codename using lsb_release -a). This will take a while:

Terminal
debootstrap cosmic /mnt/rpool/

Once done, turn off devices flag on pool and check if data has been written or we messed the paths up:

Terminal
zfs set devices=off rpool
zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 218M 29.6G 217M /mnt/rpool

Since our system is bare, we do need to prepare a few configuration files:

Terminal
cp /etc/hostname /mnt/rpool/etc/hostname
cp /etc/hosts /mnt/rpool/etc/hosts
cp /etc/netplan/*.yaml /mnt/rpool/etc/netplan/
sed '/cdrom/d' /etc/apt/sources.list > /mnt/rpool/etc/apt/sources.list

Finally we get to try our our new system:

Terminal
mount --rbind /dev /mnt/rpool/dev
mount --rbind /proc /mnt/rpool/proc
mount --rbind /sys /mnt/rpool/sys
chroot /mnt/rpool/ /bin/bash --login

Once in our new OS, a few further updates are in order:

Terminal
apt update

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we need to install linux image and headers:

Terminal
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Then we configure booting ZFS:

Terminal
apt install --yes zfs-initramfs
echo UUID=$(blkid -s UUID -o value /dev/disk/by-id/ata_disk1-part4) /boot ext2 noatime 0 2 >> /etc/fstab
echo UUID=$(blkid -s UUID -o value /dev/disk/by-id/ata_disk2-part4) /boot2 ext2 noatime 0 2 >> /etc/fstab

And disk decryption:

Terminal
apt install --yes cryptsetup
echo "luks1 UUID=$(blkid -s UUID -o value /dev/disk/by-id/ata_disk1-part1) none luks,discard,initramfs" >> /etc/crypttab
echo "luks2 UUID=$(blkid -s UUID -o value /dev/disk/by-id/ata_disk2-part1) none luks,discard,initramfs" >> /etc/crypttab

And install grub bootloader (select both disks – not partitions!):

Terminal
apt install --yes grub-pc

Followed by update of boot environment (some errors are ok):

Terminal
update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-4.18.0-12-generic
cryptsetup: ERROR: Couldn't resolve device rpool
cryptsetup: WARNING: Couldn't determine root device

Now we update the grub and fix its config (only needed if you are not using sub-datasets):

Terminal
update-grub
sed -i "s^root=ZFS=rpool/^root=ZFS=rpool^g" /boot/grub/grub.cfg

Now we get to copy all boot files to second disk:

Terminal
cp -rp /boot/* /boot2/

With grub install we’re getting close to the end of story:

Terminal
grub-install /dev/disk/by-id/ata_disk1
Installing for i386-pc platform.
Installation finished. No error reported.

grub-install /dev/disk/by-id/ata_disk2
Installing for i386-pc platform.
Installation finished. No error reported.

Now we install full GUI and upgrade whatever needs it (takes a while):

Terminal
sudo apt-get install --yes ubuntu-desktop samba
apt dist-upgrade --yes

As this probably updated grub, we need to both correct config (only if we have bare dataset) and copy files to the other boot partition (this has to be repeated on every grub update):

Terminal
sed -i "s^root=ZFS=rpool/^root=ZFS=rpool^g" /boot/grub/grub.cfg
cp -rp /boot/* /boot2/

Having some swap is always a good idea:

Terminal
zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput -o sync=always \
-o primarycache=metadata -o secondarycache=none rpool/swap

mkswap -f /dev/zvol/rpool/swap
echo /dev/zvol/rpool/swap none swap defaults 0 0 >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

Almost there, it’s time to set root password:

Terminal
passwd

And to create our user for desktop environment:

Terminal
adduser user
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sambashare,sudo user

Finally, we can reboot (don’t forget to remove CD) and enjoy our system:

Terminal
exit
reboot

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.