A while back, I snagged a Dell R710 from Ebay for $350 delivered. This thing is a virtualization monster. 8 cores and 144GB of ECC RAM, perfect for filling with bhyve virtual machines on FreeNAS. This setup worked great for me until I became interested in Docker containers. FreeNAS just doesn’t support a docker environment, so let’s build something from the ground up and document the process.
Must use ZFS (no need for ZFS root)
Native docker support
Lightweight (no GUI, unnecessary package systems *cough cough snaps*, etc.)
With these constraints, we know we absolutely must use Linux and we would really like a distro that has openzfs in its package control. In the past I’ve used Ubuntu Server and ubuntu 16.04 has zfs available as a package, but I want to avoid snaps like the plague. Because of this, I’m going to use Debian.
First things first, we’re going to shut down the server and remove every drive but our desired boot drive. FreeNAS uses a flash drive as its root storage, but debian doesn’t quite support that. In my case, I’m going to use an old 120GB SSD in an optical drive caddy in place of the server’s disk drive. It’s easy enough to complete the guided install, and Debian has some pretty good installation literature:
I do have a few gotchas during the guided install:
Some NICs require closed-source firmware to operate. The NIC in my R710 is one of those, so I had to use the non-free debian image.
Guided partitioning failed for my install. I suspect that this is because it was trying to make a gigantic swap partition to match the 144GB of RAM. I decided not to use a swap partition and partition the entire SSD as /.
I avoided giving root a password. Instead, I have an administrator user with sudo access.
I did not install any desktop environments or X server components.
Don’t forget to select openssh!
Now that we have a fully running debian system, let’s change a few things. Pop all your drives back in and ssh in.
Let’s gear up to use ssh without a password, starting by setting up private key authentication. If you’re on linux, you’ve got it easy: $ ssh-copy-id $USER@$HOST. On Windows, I find it easiest to manually copy the contents of %HOMEPATH%\.ssh\id_rsa.pub to ~/.ssh/authorized_keys on my server. If you don’t have an ~/.ssh/id_rsa.pub or %HOMEPATH%\.ssh\id_rsa.pub, you better generate an ssh key with $ ssh-keygen.
Once your public key is copied in, it’s a good idea to exit your ssh session and attempt to login with private key auth. If you don’t need a password, you’re on the right path. Time to disable password authentication. In /etc/ssh/sshd_config, find the lines containing the following keys, uncomment them and make sure they’re set to no.
Now you can reload sshd and bask in paswordless security strong enough to leave exposed publicly. $ sudo /etc/init.d/sshd reload
It may take a little while to run the dkms build, and it may throw a few warnings. They do not specifically recommend a reboot on the Debian Wiki, but it’s never a bad idea when dealing with kernel modules.
Just for grins, if you happen to be installing on a system that used to run FreeNAS, you can try to import your old pool with $ sudo zpool import. You’re likely to see action: The pool can be imported using its name or numeric identifier and the '-f' flag.. You can force the import by running $ sudo zpool import -f $POOLNAME.
If you’re starting fresh, now is the time to make your first zfs pool. I’m in a bit of a hard drive shortage, so for testing’s sake I will be doing a stripe pool with two 500GB hard drives. ZFS uses disk IDs or paths to reference disks instead of /dev/sdX paths. You can figure out which disk is which with the following commands:
$ ls -l /dev/disk/by-id
$ ls -l /dev/disk/by-path
$ lsblk -o NAME,SIZE,MODEL,VENDOR
With the device IDs in hand, I’ll make a zpool with:
$ sudo zpool create tank ata-ST9500325AS_6VESPM0A ata-ST9500325AS_6VESRNXY
You can verify that your docker install is functional by running hello-world
$ docker run --rm hello-world
Let’s Hook up Docker and ZFS
This part’s for the adventurous. You could certainly just make a bunch of ZFS datasets with filesystem mounts and mount those in your containers for bulk storage, but that’s no fun! Docker has a ZFS storage driver that will back all container, image, and volume storage with zfs. Let’s set it up.
$ sudo /etc/init.d/docker stop
Back up /var/lib/docker just in case and then delete its contents:
Create a ZFS-backed mountpoint at /var/lib/docker. The official docker instructions suggest making a zpool and mounting it there, but I’ll use a dataset since I already have a zpool configured (name tank).
Tell Docker to use ZFS for its storage driver. In /etc/docker/daemon.json:
Start docker back up and verify you’re using the ZFS storage driver:
$ sudo /etc/init.d/docker start
$ docker info
Server Version: 19.03.11
Storage Driver: zfs
Zpool Health: ONLINE
Parent Dataset: tank/docker
Space Used By Parent: 17283463680
Space Available: 935802788864
Parent Quota: no
Logging Driver: json-file
Cgroup Driver: cgroupfs
Now we have a server running Debian with Docker using ZFS-backed storage. This would be a great way to put together a NAS setup using samba or NFS with a bunch of self-hosted services. I’ll be using Docker to host a home assistant instance with nginx sitting in front as a reverse proxy.