Minimum Brother MFC-J475DW Setup on Ubuntu 24.04

As I switched quite a lot of my daily work to Linux, I kept printing and scanning to the Windows. It's not that my Brother MFC-J475DW had no Linux drivers - they are available. However, since the printer is quite an old beast now, you'll notice that the official instructions call for installing i386 packages. Guess what I don't want on my system?

So, I used my new installation as an opportunity to adjust procedure a bit and do as minimal setup as possible.

When it comes to printer packages, there is some good news. While packages do identify as 32-bit, they are actually agnostic and they don't actually need i386 architecture installed. We just use dpkg to install them:

sudo dpkg -i --force-all mfcj475dwlpr-3.0.0-1.i386.deb
sudo dpkg -i --force-all mfcj475dwcupswrapper-3.0.0-1.i386.deb

With those, your printer is installed, assuming you're happy with the default settings. You can access CUPS interface at http://localhost:631/printers and adjust it further from there. However, since I use it via a network, this will not work. I need to adjust its IP address (the easiest way to do it is by deleting and recreating the printer):

lpadmin -x MFCJ475DW
lpadmin -p Inkjet -D "Brother MFC-J475DW" -E \
        -v lpd://<IP>/binary_p1 \
        -P /opt/brother/Printers/mfcj475dw/cupswrapper/brother_mfcj475dw_printer_en.ppd

Here I also like to adjust default resolution to be "Best" as I only use my inkjet for photos anyhow:

lpoptions -p Inkjet  -l
lpadmin -p Inkjet -o BRResolution=Best

With printer out of the way, we can follow the same principle to install the scanner driver:

sudo dpkg -i --force-all brscan4-0.4.11-1.amd64.deb
sudo brsaneconfig4 -a name=Inkjet model=MFC-J475DW ip=<IP>

And that's all there is to it.


PS: These instructions work for pretty much any Brother printer.

Ubuntu 24.04 ZFS Mirror on Framework 16 Laptop (with Hibernate)

ZFS was, and still is, the primary driver for my Linux adventures. Be it snapshots or seamless data restoration, once you go ZFS it's really hard to go back. And to get the full benefits of ZFS setup you need at least two drives. Since my Framework 16 came with two drives, the immediate idea was to setup ZFS mirror.

While Framework 16 does have two NVMe drives, they are not the same. One of them is full-size M.2 2280 slot and that's the one I love. The other one is rather puny 2230 in size. Since M.2 2230 SSDs are limited to 2 TB in size, that also puts an upper limit on our mirror size. However, I still decided to combine it with a 4 TB drive.

My idea for setup is as follows: I match the smaller drive partitioning exactly so I can have myself as much mirrored disk space as possible. Leftover space I get to use for files that are more forgiving when it comes to a data loss.

I also wanted was a full disk encryption using LUKS (albeit most of the steps work if you have native ZFS encryption too). Since this is a laptop, I definitely wanted hibernation support too as it makes life much easier.

Now, easy and smart approach might be to use Ubuntu's ZFS installer directly and let it sort everything out. And let nobody tell you anything is wrong with that. However, I personally like a bit more controlled approach that requires a lot of manual steps. And no, I don't remember them by heart - I just do a lot of copy/paste.

With that out of the way, let's go over the necessary steps.

The first step is to boot into the "Try Ubuntu" option of the USB installation. Once we have a desktop, we want to open a terminal. And, since all further commands are going to need root access, we can start with that.

sudo -i

Next step should be setting up a few variables - disk, pool name, hostname, and username. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system.

DISK1=/dev/disk/by-id/<disk1>
DISK2=/dev/disk/by-id/<disk2>
HOST=<hostname>
USERNAME=<username>

On a smaller drive I wanted 3 partitions. The first two partitions are unencrypted and in charge of booting. While I love encryption, I almost never encrypt the boot partition in order to make my life easier as you cannot seamlessly integrate the boot partition password prompt with the later password prompt thus requiring you to type the password twice (or thrice if you decide to use native ZFS encryption on top of that). Third partition would be encrypted and take the rest of the drive.

On bigger drive I decided to have 5 partitions. First three would match the smaller drive. Fourth partition is 96 GB swap in order to accommodate full the worst case scenario. Realistically, even though my laptop has 96 GB of RAM, I could have gone with a smaller swap partition but I decided to reserve this space for potential future adventures. The last partition will be for extra non-mirrored data.

All these requirements come in the following few partitioning commands:

DISK1_LASTSECTOR=$(( `blockdev --getsz $DISK1` / 2048 * 2048 - 2048 - 1 ))
DISK2_LASTSECTOR=$(( `blockdev --getsz $DISK2` / 2048 * 2048 - 2048 - 1 ))

blkdiscard -f $DISK1 2>/dev/null
sgdisk --zap-all                                 $DISK1
sgdisk -n1:1M:+127M            -t1:EF00 -c1:EFI  $DISK1
sgdisk -n2:0:+1920M            -t2:8300 -c2:Boot $DISK1
sgdisk -n3:0:$DISK1_LASTSECTOR -t3:8309 -c3:LUKS $DISK1
sgdisk --print                                   $DISK1

PART1UUID=`blkid -s PARTUUID -o value $DISK1-part1`
PART2UUID=`blkid -s PARTUUID -o value $DISK1-part2`

blkdiscard -f $DISK2 2>/dev/null
sgdisk --zap-all                                                $DISK2
sgdisk -n1:1M:+127M            -t1:EF00 -c1:EFI  -u1:$PART1UUID $DISK2
sgdisk -n2:0:+1920M            -t2:8300 -c2:Boot -u2:$PART2UUID $DISK2
sgdisk -n3:0:$DISK1_LASTSECTOR -t3:8309 -c3:LUKS -u3:R          $DISK2
sgdisk -n4:0:+96G              -t4:8200 -c4:Swap -u4:R          $DISK2
sgdisk -n5:0:$DISK2_LASTSECTOR -t5:8309 -c5:LUKS                $DISK2
sgdisk --print                                                  $DISK2

And yes, using the same partition UUIDs for boot drives is important and we'll use it later to have a mirror of our boot data too.

The next step is to setup all LUKS partitions. If you paid attention, that means we need to repeat formatting a total of 4 times. Unless you want to deal with multiple password prompts, make sure to use the same password for each:

cryptsetup luksFormat -q --type luks2 \
    --sector-size 4096 \
    --perf-no_write_workqueue --perf-no_read_workqueue \
    --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf argon2i $DISK1-part3

cryptsetup luksFormat -q --type luks2 \
    --sector-size 4096 \
    --perf-no_write_workqueue --perf-no_read_workqueue \
    --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf argon2i $DISK2-part3

cryptsetup luksFormat -q --type luks2 \
    --sector-size 4096 \
    --perf-no_write_workqueue --perf-no_read_workqueue \
    --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf argon2i $DISK2-part4

cryptsetup luksFormat -q --type luks2 \
    --sector-size 4096 \
    --perf-no_write_workqueue --perf-no_read_workqueue \
    --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf argon2i $DISK2-part5

Since creating encrypted partitions doesn't mount them, we do need this as a separate step. I like to name my LUKS devices based on partition names so we can recognize them more easily:

cryptsetup luksOpen $DISK1-part3 ${DISK1##*/}-part3
cryptsetup luksOpen $DISK2-part3 ${DISK2##*/}-part3
cryptsetup luksOpen $DISK2-part4 ${DISK2##*/}-part4
cryptsetup luksOpen $DISK2-part5 ${DISK2##*/}-part5

Finally, we can set up our mirrored ZFS pool with an optional step of setting quota to roughly 85% of disk capacity. Since we're using LUKS, there's no need to setup any ZFS keys. Name of the mirrored pool will match name of the host and it will contain several datasets to start with. It's a good starting point, adjust as needed:

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O quota=1600G \
    -O canmount=off -O mountpoint=none -R /mnt/install \
    ${HOST^} mirror /dev/mapper/${DISK1##*/}-part3 /dev/mapper/${DISK2##*/}-part3

zfs create -o canmount=noauto -o mountpoint=/ \
    -o reservation=100G \
    ${HOST^}/System
zfs mount ${HOST^}/System

zfs create -o canmount=noauto -o mountpoint=/home \
           ${HOST^}/Home
zfs mount ${HOST^}/Home
zfs set canmount=on ${HOST^}/Home

zfs create -o canmount=noauto -o mountpoint=/Data \
           ${HOST^}/Data
zfs set canmount=on ${HOST^}/Data

zfs set devices=off ${HOST^}

Of course, we can also setup our extra non-mirrored pool:

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O quota=1600G \
    -O canmount=on -O mountpoint=/Extra \
    ${HOST^}Extra /dev/mapper/${DISK2##*/}-part5

With ZFS done, we might as well setup boot, EFI, and swap partitions too. Any yes, we don't have mirrored boot and EFI at this time; we'll sort that out later.

yes | mkfs.ext4 $DISK1-part2
mkdir /mnt/install/boot
mount $DISK1-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI -i 4d65646f $DISK1-part1
mkdir /mnt/install/boot/efi
mount $DISK1-part1 /mnt/install/boot/efi

mkswap /dev/mapper/${DISK2##*/}-part4

At this time, I also sometimes disable IPv6 as I've noticed that on some misconfigured IPv6 networks it takes ages to download packages. This step is both temporary (i.e., IPv6 is disabled only during installation) and fully optional.

sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
sysctl -w net.ipv6.conf.lo.disable_ipv6=1

To start the fun we need to debootstrap our OS. As of this step, you must be connected to the Internet.

apt update
apt dist-upgrade --yes
apt install --yes debootstrap
debootstrap noble /mnt/install/

We can use our live system to update a few files on our new installation:

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
rm /mnt/install/etc/apt/sources.list
cp /etc/apt/sources.list.d/ubuntu.sources /mnt/install/etc/apt/sources.list.d/ubuntu.sources
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

At last, we're ready to chroot into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install /usr/bin/env \
    DISK1=$DISK1 DISK2=$DISK2 HOST=$HOST USERNAME=$USERNAME \
    bash --login

With our newly installed system running, let's not forget to set up locale and time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

ln -sf /usr/share/zoneinfo/America/Los_Angeles /etc/localtime
dpkg-reconfigure -f noninteractive tzdata

Now we're ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends \
    linux-image-generic linux-headers-generic

Now we set up crypttab so our encrypted partitions are decrypted on boot.

echo "${DISK1##*/}-part3 $DISK1-part3 none \
      luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
echo "${DISK2##*/}-part3 $DISK2-part3 none \
      luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab

echo "${DISK2##*/}-part4 $DISK2-part4 none \
      luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab

echo "${DISK2##*/}-part5 $DISK2-part5 none \
      luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab

cat /etc/crypttab

To mount all those partitions, we also need some fstab entries. ZFS entries are not strictly needed. I just like to add them in order to hide our LUKS encrypted ZFS from the file manager:

echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK1-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=3s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK1-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=3s 0 1" >> /etc/fstab

echo "/dev/mapper/${DISK1##*/}-part3 \
    none auto nofail,nosuid,nodev,noauto 0 0" >> /etc/fstab
echo "/dev/mapper/${DISK2##*/}-part3 \
    none auto nofail,nosuid,nodev,noauto 0 0" >> /etc/fstab

echo "/dev/mapper/${DISK2##*/}-part4 \
    swap swap nofail 0 0" >> /etc/fstab

echo "/dev/mapper/${DISK2##*/}-part5 \
    none auto nofail,nosuid,nodev,noauto 0 0" >> /etc/fstab

cat /etc/fstab

On systems with a lot of RAM, I like to adjust memory settings a bit. This is inconsequential in the grand scheme of things, but I like to do it anyway.

echo "vm.swappiness=10" >> /etc/sysctl.conf
echo "vm.min_free_kbytes=1048576" >> /etc/sysctl.conf

Now we can create the boot environment:

apt install --yes zfs-initramfs cryptsetup keyutils grub-efi-amd64-signed shim-signed
update-initramfs -c -k all

And then, we can get grub going. Do note we also set up booting from swap (needed for hibernation) here too. If you're using secure boot, bootloaded-id HAS to be Ubuntu.

apt install --yes grub-efi-amd64-signed shim-signed
sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash \
    RESUME=UUID=$(blkid -s UUID -o value /dev/mapper/${DISK2##*/}-part4)\"/" \
    /etc/default/grub
update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi \
    --bootloader-id=Ubuntu --recheck --no-floppy

I don't like snap so I preemptively banish it from ever being installed:

apt remove --yes snapd 2>/dev/null
echo 'Package: snapd'    > /etc/apt/preferences.d/snapd
echo 'Pin: release *'   >> /etc/apt/preferences.d/snapd
echo 'Pin-Priority: -1' >> /etc/apt/preferences.d/snapd
apt update

And now, finally, we can install our desktop environment.

apt install --yes ubuntu-desktop-minimal man

Since Firefox is a snapd package (banished), we can install it manually:

add-apt-repository --yes ppa:mozillateam/ppa
cat << 'EOF' | sed 's/^    //' | tee /etc/apt/preferences.d/mozillateamppa
    Package: firefox*
    Pin: release o=LP-PPA-mozillateam
    Pin-Priority: 501
EOF
apt update && apt install --yes firefox

Chrome aficionados, can install it too:

pushd /tmp
wget --inet4-only https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
apt install ./google-chrome-stable_current_amd64.deb
popd

If you still remember the start of this post, we are yet to mirror our boot and EFI partition. For this, I have a small utility we might as well install now:

wget -O- http://packages.medo64.com/keys/medo64.asc | sudo tee /etc/apt/trusted.gpg.d/medo64.asc
echo "deb http://packages.medo64.com/deb stable main" | sudo tee /etc/apt/sources.list.d/medo64.list
apt update
apt install -y syncbootpart
syncbootpart

With Framework 16, there are no mandatory changes you need to do in order to have the system working. That said, I still like to do a few changes; the first of them is to allow trim operation on expansion cards:

cat << EOF | tee /etc/udev/rules.d/42-framework-storage.rules
ACTION=="add|change", SUBSYSTEM=="scsi_disk", ATTRS{idVendor}=="13fe", ATTRS{idProduct}=="6500", ATTR{provisioning_mode}:="unmap"
ACTION=="add|change", SUBSYSTEM=="scsi_disk", ATTRS{idVendor}=="32ac", ATTRS{idProduct}=="0005", ATTR{provisioning_mode}:="unmap"
ACTION=="add|change", SUBSYSTEM=="scsi_disk", ATTRS{idVendor}=="32ac", ATTRS{idProduct}=="0010", ATTR{provisioning_mode}:="unmap"
EOF

Since we're doing hibernation, we might as well disable some wake up events that might interfere. I explain the exact process in another blog post but suffice it to say, this works for me:

cat << EOF | sudo tee /etc/udev/rules.d/42-disable-wakeup.rules
ACTION=="add", SUBSYSTEM=="i2c", DRIVER=="i2c_hid_acpi", ATTRS{name}=="PIXA3854:00", ATTR{power/wakeup}="disabled"
ACTION=="add", SUBSYSTEM=="pci", DRIVER=="xhci_hcd", ATTRS{subsystem_device}=="0x0001", ATTRS{subsystem_vendor}=="0xf111", ATTR{power/wakeup}="disabled"
ACTION=="add", SUBSYSTEM=="serio", DRIVER=="atkbd", ATTR{power/wakeup}="disabled"
ACTION=="add", SUBSYSTEM=="usb", DRIVER=="usb", ATTR{power/wakeup}="disabled"
EOF

For hibernation I like to change sleep settings so that hibernation kicks in after 13 minutes of sleep:

sed -i 's/.*AllowSuspend=.*/AllowSuspend=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*AllowHibernation=.*/AllowHibernation=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*AllowSuspendThenHibernate=.*/AllowSuspendThenHibernate=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*HibernateDelaySec=.*/HibernateDelaySec=13min/' \
    /etc/systemd/sleep.conf

For that we also need to do a minor lid switch configuration adjustment:

apt install -y pm-utils

sed -i 's/.*HandlePowerKey=.*/HandlePowerKey=hibernate/' \
    /etc/systemd/logind.conf
sed -i 's/.*HandleLidSwitch=.*/HandleLidSwitch=suspend-then-hibernate/' \
    /etc/systemd/logind.conf
sed -i 's/.*HandleLidSwitchExternalPower=.*/HandleLidSwitchExternalPower=suspend-then-hibernate/' \
    /etc/systemd/logind.conf

Lastly, we need to have a user too.

adduser --disabled-password --gecos '' $USERNAME
usermod -a -G adm,cdrom,dialout,dip,lpadmin,plugdev,sudo,tty $USERNAME
echo "$USER ALL=NOPASSWD:ALL" > /etc/sudoers.d/$USERNAME
passwd $USERNAME

It took a while, but we can finally exit our debootstrap environment:

exit

Let's clean all mounted partitions and get ZFS ready for next boot:

sync
umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

After reboot, we should be done and our new system should boot with a password prompt.

reboot

Once we log into it, I like to first increase text size a bit:

gsettings set org.gnome.desktop.interface text-scaling-factor 1.25

Now we can also test hibernation:

sudo systemctl hibernate

If you get Failed to hibernate system via logind: Sleep verb "hibernate" not supported, go into BIOS and disable secure boot (Enforce Secure Boot option). Unfortunately, the secure boot and hibernation still don't work together but there is some work in progress to make it happen in the future. At this time, you need to select one or the other.

Assuming all works nicely, we can get firmware updates going:

fwupdmgr enable-remote -y lvfs-testing
fwupdmgr refresh
fwupdmgr update

And that's it - just half a thousand steps and you have Ubuntu 24.04 with a ZFS mirror.

Powering External Devices while Protected

Boards for this project were sponsored by PCBWay.

I have designed a few framework expansion cards already, but I tried to keep them data-only interface. That is, I avoided providing power as much as possible. And, in my opinion, that's the proper way of doing it. However, what if I occasionally want to provide some power too? Can it be done (reasonably) safely?

Considering my intentions of using an expansion card, and thus sourcing power directly from the laptop, my goals were to avoid the two most common issues: current backflow and current limiting.

While using fuses for current limiting is a traditionally accepted method, it's neither precise nor fast enough to protect a laptop. Fortunately, there are load switches that allow for not only precise current limiting but also for fast response.

My choice fell onto AP22652Wx family. These devices allow for quite precise current limiting using external resistor, short-circuit and reverse voltage/current blocking. All that in a small SOT26 (aka SOT23-6) package.

As it often goes, we do need a few external components too - at least one resistor (to limit current) and at least 100 nF decoupling capacitor. Realistically, if you're not sure what gets connected on the other side, you need a few more capacitors. I usually opt for 100 nF/10 μF combo on both input and output. And here is the biggest disadvantage of using load switch - quite a lot of PCB real estate.

Second issue I mentioned (current "backflow") is also known as reverse current. This happens when one accidentally connect a positive voltage to a power output. As often in life, bigger force "wins" and thus current will start flowing into the output pin. When that device is a framework expansion card powered by laptop
itself, this is definitely not the most comfortable situation.

Load switch I selected does offer basic reverse-current protection, but I wanted a bit of a foolproof solution and that comes in the form of a humble diode. Such forward biased diode will allow current to flow towards output but it will stop conducting if output voltage goes higher than that. And yes, that comes at a cost of voltage drop. However, if you keep current low, voltage drop is still within a few hundred millivolts.

In my case I decided to limit current to 400 mA and thus CUS08F30 I selected has not only healthy headroom with current but it would drop under 400 mV at a full load. Yes, there are diodes that have even better characteristics but this one comes in quite a small package and I already had it lying around.

With theory solved, it was time to test it. For that purpose I've created a new Power framework expansion card https://github.com/medo64/PowerFEC. I used PIC16F1454 to interface it to USB, small ZXCT1009 to measure current, in addition to load switch and diode described above.

PCBs were manufactured courtesy of PCBWay. When it comes to framework expansion cards, they became my go-to manufacturer for two reasons. The first one is their standard routing bit seems to be much smaller than the one used by competition. This is important when it comes to the Type-C edge connector which requires a really small feature that large routing bits will mangle. The second nice thing is that PCBWay doesn't use break-away connections, opting instead for V-groove. This means the edge is as perfect as it gets, saving me from manual sanding.

And yes, they're not perfect. They still put their screen markings (you can pay extra to keep it clean). They still use leaded HASL (you can pay extra for non-leaded or ENIG). And sometimes waiting for somebody to manually check PCBs really clashes with my habit of finishing PCBs just before my bedtime thus forcing me to wait (albeit never too long).

But, all in all, those are all things that are minor enough not to bother me too much. I haven't had bad PCBs from them in ages. Quite often with small PCBs I would get a few more than I ordered. And HASL is fine for pretty much all my use cases. While this PCB was sponsored, I used PCBWay on my own dime without complaint.

In order to make it all work, I decided to reuse my old code and make device behave as a serial port. It will output voltage and current in addition to allowing for a few basic commands (e.g. enable and disable).

After messing with it quite a while, I found protection working pretty much perfectly, thus expect a few updated expansion card designs in store.

As always, hardware and software are available on GitHub

Recovering ZFS

Well, after using ZFS for years, it was only a matter of time before I encountered an issue. It all started with me becoming impatient with my Ubuntu 20.04 desktop. The CPU was at 100% and the system was actively writing to disk (courtesy of ffmpeg), but I didn't want to wait. So, I decided to do a hard reset. What's the worst that could happen?

Well, boot process took a while and I was presented with bunch of entries looking like this:

INFO: task z_wr_iss blocked for more than 122 seconds.
      Tainted: P      0E      6.8.0-35-generic #35-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
INFO: task z_metaslab blocked for more than 122 seconds.
      Tainted: P      0E      6.8.0-35-generic #35-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
INFO: task vdev_autotrim blocked for more than 122 seconds.
      Tainted: P      0E      6.8.0-35-generic #35-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
…

At first, I thought the system would recover on its own as this wasn't the first time I had mistreated it. However, leaving it alone did nothing. So, it was time for recovery.

The first step was getting to the GRUB menu. Pressing <ESC> multiple times during boot simply dropped me to a prompt. And yes, you can load initramfs manually from there, but it's a bit tedious. However, in this case, I just typed normal to get the menu, followed by <E> to edit the line starting with "linux". There, I appended break, telling the system to drop me into initramfs prompt so that I could manually load ZFS.

From here, there was another hurdle to bypass. While this stopped before ZFS was loaded, it also stopped before my disks were decrypted. Had I used native ZFS encryption, this wouldn't be a problem, but I wanted LUKS, so now I had to load them manually. As I had a mirror, I used the following to open both:

DISK1=/dev/disk/by-id/nvme-<disk1>
DISK2=/dev/disk/by-id/nvme-<disk2>

cryptsetup luksOpen $DISK1-part4 ${DISK1##*/}-part4
cryptsetup luksOpen $DISK2-part4 ${DISK2##*/}-part4

Finally, I was ready to import the pool and decided to do it in read-only mode:

zpool import -o readonly=on <pool>

And surprisingly, it worked. That also meant that my recovery efforts didn't need to go too far. So, I decided to try importing it again but in read/write mode:

zpool export <pool>
zpool import <pool>

And then I was greeted with an ominous message:

PANIC: zfs: adding existent segment to range tree

However, the import didn't get stuck as such, and my data was still there. So, I decided to give it a good scrub:

zpool scrub <pool>

While the scrub didn't find any errors, going over all data seemed to have resulted in data structures "straightening out" and thus everything looked as good as before.

One reboot later, and I got into my desktop just fine.


PS: If that failed, I would have probably gone with zpool import -F <pool>.

PPS: If that also failed, disabling replays would be my next move.

echo 1 > /sys/module/zfs/parameters/zil_replay_disable
echo 1 > /sys/module/zfs/parameters/zfs_recover

PPPS: You can also add those parameters to "linux" grub line (zfs.zil_replay_disable=1 zfs.zfs_recovery=1).