Automatically and securely mounting encrypted ZFS filesystems at boot with Azure Key Vault

  • 2 min read

    • The need for automation

    As noted in my prior blogs, I use ZFS on Linux for my home fileserver and have been very impressed - it’s been extremely stable, versatile and the command line utilities have simple syntax that work exactly as you’d expect them to.

    A few months back native encryption was introduce into master branch for testing (you can read more here), and I have been using it to encrypt all my data. I chose not encrypt my root drive since it doesn’t host any user data, and I do not want my boot to be blocked on password input - for example what if there’s a power failure while I’m travelling for work?

    However that still leaves two nagging problems:

    1. It became tedious to manually SSH into my machine every time it restarts to type in numerous encrypted filesystem passphrases
    2. A bunch of my systemd services depend on user data; issue in systemd (#8587) prevents using auto-generated mount dependenices to wait for the filesystems to be mounted so I have to start them menually.

    Introducing zfs-keyvault

    I decided to kill two birds with one stone and am happy to introduce zfs-keyvault, available on GitHub. It provides both a systemd service that can be depended upon by other services, as well automation for securely mounting encrypted ZFS filesystems.

    On the client (with ZFS filesystems), a zkv utility is installed that can be used to manage an encrypted repository containing one or more ZFS filesystem’s encryption keys. This repository is locally stored and its encryption key is placed in an Azure Key Vault.

    On your preferred webhost or public cloud, a small Flask webserver called zkvgateway gates access to this repository key in Key Vault and can release under certain conditions.

    On boot, the systemd service runs zkv which will reach out to the gateway, who in turn SMSs you with a PIN for approval. The inclusion of a PIN stops people from blindly hitting your endpoint to approve requests, and also prevents replay attacks. The gateway is also rate-limited to 1 request/s to stop brute-force attacks.

    Once the PIN is confirmed over SMS, repository key is released from Azure Key Vault and the zkv utility can now decrypts the ZFS filesystem encryption keys which are locally stored, and begins mounting the filesystems. The filesystem encryption keys never leave your machine!

    I’ve uploaded the server-side components as a Docker image named stewartadam/zkvgateway so it can be pulled and run easily. Enjoy!