VirtualBox Host I/O Cache

My XigmaNAS-based file server is usually quite a speedy beast. Between 8 disk ZFS RAID-Z2 and LACP, it can pretty much handle everything I need for my home. Except VirtualBox.

When I tried using its built-in VirtualBox, the guest virtual machine was really slow to install. Disk transfer was in kilobytes. And it wasn’t the disk speed problem as I could copy files in the background at the excess of 100 MB/s. After a bit of investigation, culprit was found in the way how VirtualBox writes to disk. Every write is essentially flushed. Combine that with ZFS on spinning rust and you have ridiculously low performance.

There are essentially two ways to solve this. The first one is to enable such pattern on ZFS. Adding logging SSD disk to my array would do wonders. However, considering this was the only load requiring them, I didn’t want to go through neither the cost or the effort of setting mirrored logging devices.

Another fix is much easier and comes without the cost. I just enabled Use Host I/O Cache for my virtual controller and speed went through the roof. Yes, this solution makes host crashes really dangerous as all cached data will be lost. And that’s a few seconds worth of important guest file system data with a potential to cause corruption. You should really think twice before turning it on.

However, for my VM it proved to be good enough. All data used by that VM lived on network shares to start with and recovering from corrupted OS didn’t bother me much as I scripted the whole setup anyhow.

Low cost solution for when you can handle a data loss potential.

Bimil failing with FontFamilyNotFound

While Bimil is primarily Windows application, I use it regularly on Linux. However, when I tried running it on freshly installed Linux Mint 19.3, I was greeted with a quick crash.

Running it from console did shine a bit more light onto the situation as the following line was quite noticeable: [ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentException: The requested FontFamily could not be found [GDI+ status: FontFamilyNotFound].

As mono is Windows Forms application written in C#, running it on Linux requires a few packages extra. Most of them are actually taken care of with the installation of mono-complete. However, in Linux mint, I found one other dependency I was not aware of – Microsoft fonts.


sudo apt-get install ttf-mscorefonts-installer

Using Let’s Encrypt with Certificate Based Authentication

For one of my sites I wanted to use TLS client authentication. It’s easy enough to setup in Apache:

<VirtualHost *:80>

RewriteEngine On
RewriteRule (.*) https://%{SERVER_NAME}$1 [R=301,L]

<VirtualHost *:443>

SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/
SSLCertificateKeyFile /etc/letsencrypt/live/
SSLCertificateChainFile /etc/letsencrypt/live/
SSLVerifyClient require
SSLVerifyDepth 1
SSLCACertificateFile /srv/apache/data/root.crt
SSLRequire (%{SSL_CLIENT_S_DN_CN} == "Me") \
|| (%{SSL_CLIENT_S_DN_CN} == "Myself") \
|| (%{SSL_CLIENT_S_DN_CN} == "Irene")

And this worked just fine for 90 days or so. More precisely, it worked until CertBot had to update my Let’s Encrypt certificate.

Guess what? Let’s Encrypt doesn’t have knowledge of my client certificate and thus handshake fails. Error message is not really helpful as “tls: unexpected message” doesn’t really point you to the correct path. Fortunately, I actually remembered my certificate shenanigans and thus was able to debug it quite quickly. Issue verification was as easy as dropping certificate requirements made my renewal work again.

However, dropping certificates every month or two would not work for me. I wanted something that would work the same as automatic renewal for other Let’s Encrypt certificates. And no, you cannot set .well-known directory to use different validation. With TLS 1.3, you cannot change client requirements once connection is established. You’ll just get “Cannot perform Post-Handshake Authentication” error.

But, you know where you can play with locations to your heart’s content? In HTTP section. Instead of just redirecting to HTTPS, you want to carve small hole for CertBot verification.

<VirtualHost *:80>

RewriteEngine On
RewriteRule (.*) https://%{SERVER_NAME}$1 [R=301,L]
<Location "/.well-known/">
RewriteEngine Off

Now Let’s Encrypt verifies renewal requests using HTTP which is not really a security issue as verification file is completely random and generated anew each time.

VPN-only Internet Access on Linux Mint 19.3 via Private Internet Access

Setting up Private Internet Access VPN is usually not a problem these days as Linux version is readily available among the supported clients. However, such installation requires GUI. What if we don’t want or need one?

For setup to work independently of GUI, one approach is to use OpenVPN client usually installed by default. Also needed are PIA’s IP-based OpenVPN configuration files. While this might cause issues down the road if that IP changes, it does help a lot with security as we won’t need to poke an unencrypted hole (and thus leak information) for DNS.

From the PIA configuration archive extract your choice of .ovpn file (usually going with the one physically closest to you will give you the best results). There is no need to extract .crt and .pem files as configuration has certificates embedded.

Rest of the VPN configuration needs to be done from the Bash:

sudo cp sudo cp ~/Downloads/openvpn-ip/US\ Seattle.ovpn /etc/openvpn/client/pia.conf

echo "auth-user-pass /etc/openvpn/client/pia.login" | sudo tee -a /etc/openvpn/client/pia.conf
echo "mssfix 1400" | sudo tee -a /etc/openvpn/client/pia.conf
echo "dhcp-option DNS" | sudo tee -a /etc/openvpn/client/pia.conf
echo "dhcp-option DNS" | sudo tee -a /etc/openvpn/client/pia.conf
echo "script-security 2" | sudo tee -a /etc/openvpn/client/pia.conf
echo "up /etc/openvpn/update-resolv-conf" | sudo tee -a /etc/openvpn/client/pia.conf
echo "down /etc/openvpn/update-resolv-conf" | sudo tee -a /etc/openvpn/client/pia.conf

The basic VPN setup is already completed but we still need to setup our login (replacing username and password with the actual values):

echo 'username' | sudo tee -a /etc/openvpn/client/pia.login
echo 'password' | sudo tee -a /etc/openvpn/client/pia.login
sudo chmod 400 /etc/openvpn/client/pia.login

Firewall rules are to allow data flow only via VPN’s tun0 interface with only encrypted VPN traffic being allowed on port 1198.

yes | sudo ufw reset
sudo ufw default deny incoming
sudo ufw default deny outgoing
sudo ufw allow out on tun0
sudo ufw allow out on eth0 proto udp to `cat /etc/openvpn/client/pia.conf \
| grep "^remote " | grep -o ' [^ ]* '` port 1198
sudo ufw enable

To test VPN connection execute:

sudo openvpn --config /etc/openvpn/client/pia.conf

Assuming test was successful (i.e. resulted in Initialization Sequence Completed message), we can further make sure data is actually traversing VPN. I’ve found quite helpful here. Just check if IP detected is different then IP you usually get without VPN.

Stop the test connection using Ctrl+C and proceed to configure OpenVPN’s auto-startup. Reboot is there just to test if auto-startup works.

sudo systemctl enable openvpn-client@pia
sudo reboot

This should give you quite secure setup without the need for GUI.