The very first thing I forgot last time is randomizing the disks upfront. While not increasing security of new data, it does remove any old unencrypted bits you might have laying around. Even if disk is fresh, you don’t want zeros showing where your data is. Dangerous utility called
dd comes handy here (once for each disk):
# dd if=/dev/urandom of=/dev/ada0 bs=1M # dd if=/dev/urandom of=/dev/ada1 bs=1M
This takes a while but fortunately it is possible to see current progress with
Ctrl+T. Do use
tmuxto keep session alive as this will take long time (with a big disk, more than a day is not unexpected).
Next, instead of using
glabel, I decided to use the whole disk. That makes it easier to move disk later to other platform. No, I am not jumping BSD ship but I think having setup that can change environments is really handy for emergency recovery.
While ZFS can handle using device names like
ada1 and all shenanigans that come with their dynamic order, I decided to rely on serial number of drive. Normally device labels containing serial number are found under
/dev/diskid/ directory. However, NAS4Free doesn’t have them on by default.
To turn them on, we go to
loader.conf tab. There we add
kern.geom.label.disk_ident.enable=1 and reboot. After this, once can use
/dev/diskid/* for drive identification.
Those drives I then encrypt and attach each drive:
# geli init -e AES-XTS -l 128 -s 4096 /dev/diskid/DISK-WD-WCC7KXXXXXXX # geli init -e AES-XTS -l 128 -s 4096 /dev/diskid/DISK-WD-WCC7KYYYYYYY # geli attach /dev/diskid/DISK-WD-WCC7KXXXXXXX # geli attach /dev/diskid/DISK-WD-WCC7KYYYYYYY
Finally, I can create the pool. Notice that I put quota around 80% of the total pool capacity. Not only this helps performance but it also prevents me from accidentally filling the whole pool. Dealing with CoW file system when it is completely full is something you want to avoid. And also, do not forget
# zpool create -o autoexpand=on -m none -O compression=on -O atime=off -O utf8only=on -O normalization=formD -O casesensitivity=sensitive -O quota=3T Data mirror /dev/diskid/DISK-WD-WCC7KXXXXXXX.eli /dev/diskid/DISK-WD-WCC7KYYYYYYY.eli # zdb | grep ashift ashift: 12
Once pool was created, I snapshotted each dataset on old machine and sent it over network. Of course, this assumes your pool is named
Data, you are working from “old” machine, and new machine is at
# zfs snapshot -r Data@Migration # zfs send -Rv Data@Migration | ssh 192.168.1.2 zfs receive -Fs Data
This step took a while (more than a day) as all datasets had to be recursively sent. Network did die a few times but resumable send saved my ass.
First I would get token named
receive_resume_token from the destination:
# zfs get receive_resume_token
And resume sending with:
# zfs send -v -t <token> | ssh 192.168.1.2 zfs receive -Fs Data/dataset
Unfortunately resume token does not work with recursion so each dataset will have to be separately specified from that moment onward.
Once bulk of migration was done, I shut every single service on old server. After that I took another (much smaller) snapshot and sent it over network:
# zfs snapshot -r Data@MigrationFinal # zfs send -Ri Data@Migration Data@MigrationFinal | ssh 192.168.1.2 zfs receive -F Data
And that is it – shutdown the old machine and bring services up on the new one.
PS: If newly created machine goes down, it is enough to re-attach GELI disks followed by restart of ZFS daemon:
# geli attach /dev/diskid/DISK-WD-WCC7KXXXXXXX # geli attach /dev/diskid/DISK-WD-WCC7KYYYYYYY # /etc/rc.d/zfs onestart