Automatically and securely mounting encrypted ZFS filesystems at boot with Azure Key Vault

The need for automation

As noted in my prior blogs, I use ZFS on Linux for my home fileserver and have been very impressed - it's been extremely stable, versatile and the command line utilities have simple syntax that work exactly as you'd expect them to.

A few months back native encryption was introduce into master branch for testing (you can read more here), and I have been using it to encrypt all my data. I chose not encrypt my root drive since it doesn't host any user data, and I do not want my boot to be blocked on password input - for example what if there's a power failure while I'm travelling for work?

However that still leaves two nagging problems:
1. It became tedious to manually SSH into my machine every time it restarts to type in numerous encrypted filesystem passphrases
2. A bunch of my systemd services depend on user data; issue in systemd (#8587) prevents using auto-generated mount dependenices to wait for the filesystems to be mounted so I have to start them menually.

Introducing zfs-keyvault

I decided to kill two birds with one stone and am happy to introduce zfs-keyvault, available on GitHub. It provides both a systemd service that can be depended upon by other services, as well automation for securely mounting encrypted ZFS filesystems.

On the client (with ZFS filesystems), a zkv utility is installed that can be used to manage an encrypted repository containing one or more ZFS filesystem's encryption keys. This repository is locally stored and its encryption key is placed in an Azure Key Vault.

On your preferred webhost or public cloud, a small Flask webserver called zkvgateway gates access to this repository key in Key Vault and can release under certain conditions.

On boot, the systemd service runs zkv which will reach out to the gateway, who in turn SMSs you with a PIN for approval. The inclusion of a PIN stops people from blindly hitting your endpoint to approve requests, and also prevents replay attacks. The gateway is also rate-limited to 1 request/s to stop brute-force attacks.

Once the PIN is confirmed over SMS, repository key is released from Azure Key Vault and the zkv utility can now decrypts the ZFS filesystem encryption keys which are locally stored, and begins mounting the filesystems. The filesystem encryption keys never leave your machine!

I've uploaded the server-side components as a Docker image named stewartadam/zkvgateway so it can be pulled and run easily. Enjoy!

Securing your online identity

Within the past month or so, I find it odd that the occurrence of friends who have had their Hotmail, Gmail, Facebook or other online accounts hijacked has roughly tripled... Probably some new social engineering technique or malware is behind it, but I took the time to write this up since preventing account hijacking from happening to you is easy and probably won't take more than 5 minutes to make the changes to your online accounts.

Choose secure passwords

Part of basic online security is choosing a good password. Remember that many password crackers are loaded with dictionary words and common variations thereof (replacing a letter with a number, adding numbers after a word) - choosing a simple password makes your account is an easy target. Any secure password should be at least 8 characters in length, include letters (upper and lower case), numbers and even punctuation when possible. Below is a quick 5-step process for choosing a secure and memorable password:

  • Choose any word
  • Add two numbers to the end of the word at random
  • After the two numbers, add a punctuation mark or symbol such as: ! @ # $ % & * ( ) _ + - = [ ] \ { } | ; ' : " , . / ?
  • After the punctation mark, add another word related to the first
  • Choose one or more letters in your password and make them uppercase

For example, I like chocolate ice cream. I choose the number 38 at random, used an exclamation mark and made any letter "c" my password uppercase. The end result is ChoColate38!iCeCeam. It isn't very difficult to remember, but that would be extremely tricky to guess!

Do not answer any "secret question" with the correct answer.

Many websites (and even some banks) employ the "secret question" technique to verify your identity. Often, you can also reset your password by giving the answer to one of your secret questions. If you have a secret question that's easy to guess, having a secure password is moot; the secret question bypasses it completely. Even worse, the attacker could change your password once they break into your account locking you out!

The answer to this problem is to choose something unrelated to the question and use that for the answer. Make it obscure enough so that you can use the same answer all the time so you don't have to remember which nonsensical answer you choose for which question... For example:
What is your best friend's name? Spoon43.
What is your favourite food? Spoon43.
What was the name of the first street you lived on? Spoon43.
What is your mother's maiden name? Spoon43.

Be conscious of what you post publicly

Be careful of what you post online. Nobody thinks identity theft could ever happen to them, but happens much more often than you would expect. As well, with the rise in popularity of social networks it has become easier and easier to track people down and lift information from profiles.

  • Tighten down your privacy settings. Is there really a need to let everyone know about your personal life? Hide information that you don't want the world to see - your cell phone and home address are good examples. In Facebook's case, you also want to set all privacy settings to "Friends only".
  • Think about what you publish before you hit the button. As a general rule, don't publish to social networking sites or your blog what you wouldn't want everybody to know about you.

Do not store sensitive information without encryption

Encryption is a technique that turns information unreadable to anyone without a the key or passphrase. You should only store your sensitive data if it is encrypted as it will make it much more difficult for hackers to get at:

  • Mac OS X: All the tools you need are preinstalled. Open Disk Utility (in Applications > Utilities) and then select File > New > Blank disk image... from the menu. In the dialog that appears, select either the 128-bit (faster, less secure) or 256-bit (slower, more secure) AES methods.
  • Windows: Not all version of Windows support file encryption (Windows XP Home Edition doesn't, for example). However, you can download TrueCrypt for free. As of writing this, it supports Windows XP to Windows 7, both x32 and x64.

Additional information: How do the hackers get in?

There are many ways for them to break into your account. I've explained some of the common methods below:


Keyloggers are a type of computer malware that people often call "a virus," but it is very different from your ordinary virus. Keyloggers infect your computer and show no symptoms; they will not make your computer slower, delete your document or crash programs. Instead, they record what you type on the keyboard and send it to the hacker who wrote the keylogger. For hackers, this is a very appealing method since it has the potential to not only catch your passwords, but also your credit card numbers, online banking PINs and more.

Malwarebytes Anti-Malware is a free tool that is extremely effective at removing malicious software from your computer. The scan only takes 10 or 15 minutes, so I recommend running a scan at least once a week. Remember to update (click the "Update" tab) before running a scan!

Social engineering

To quote Wikipedia, social engineering is "the act of manipulating people into performing actions or divulging confidential information, rather than by breaking in or using technical hacking techniques." The spam messages you get from a fake "PayPal" or "Your online bank" asking you to visit some random site and enter your credit card and password to confirm your account are a good example of very basic social engineering. Since social engineering is based on sneaky and dirty tricks, it is also the hardest to prevent since there is no real solution other than keeping a watchful eye.

Brute force attacks

Brute force attacks break passwords by trying different combinations over and over again until it finds the right one. This is why having simple passwords based on dictionary words are bad, since it would be a trivial task to find by brute force.