Changes coming in 2017

I'm writing this quick post at the end of 2016 to wish everyone happy holidays and let you know about some changes coming in 2017.


Earlier this year, I stopped working for my company Diffingo Solutions Inc. and am excited to say that I now work at Microsoft, as can be seen on my github profile (@stewartadam). I think that the direction Microsoft has taken in the past two years has been very positive and I'm exicted to see what the future brings.

Of course, there's the mandatory note: the opinions on this blog are my own and may not represent those of my employer. However, given the technologies I interact with on a day-to-day basis, you may notice an increase in blog posts about technologies owned by my employer ;)

Comments are now disabled

As well, due to an influx of spam comments that are bypassing CAPTCHA and other automated anti-spam methods, I have disabled comments on my blog. I expect this will be the case for the forseeable future.

New blog style coming

I am likely going to convert my Drupal-based blog to one based on Markdown or other static rendering. Drupal has been great to me, but Markdown is so much easier to author in and requires 0 maintenance. As well, the rate at which I can author content has slowed so Drupal's original use case for me (maintaining Views-based lists of the various categories of howtos I was writing) no longer applies.

Using OpenVPN connection to play games while abroad using Stream's In-Home Streaming


Steam has a great (albeit, a little glitchy) feature called In-Home Streaming that allows you to stream games from running Steam clients on your local network, effectively turning your gaming PC into a little render farm and allowing you to play from low-power devices like a laptop.

With the help of OpenVPN, it's possible to enable playback of games from your home PC seamlessly while away from your home network too, provided you have a decent Internet connection. This tutorial will demonstrate how to setup an OpenVPN server on Fedora 23 with a bridged network connection in let you VPN into your home network and stream Steam games from PCs on your LAN.

To make this work, we need to use OpenVPN in bridged mode with a tap network device. Bridging the ethernet and tap interfaces will allow VPN clients to receive an IP address on the LAN's subnet.

The default (and simpler) tun devices are not bridged, and function on a separate subnet - something which will break in-home streaming. We need to be on the same LAN so that the UDP broadcast packets sent by Steam for auto-discovery will be received by the VPN clients.

Creating a network bridge

Let's start by setting up the network bridge with NetworkManager and enslaving the ethernet interface. Check the name of your active network interface by running nmcli d, and replace the value ofETH_IFACE with that name below:

nmcli con add type bridge ifname br0
nmcli c modify bridge-br0 bridge.stp no
nmcli con add type bridge-slave ifname $ETH_IFACE master bridge-br0
nmcli c up "bridge-slave-${ETH_IFACE}"
nmcli c up bridge-br0

Installing the OpenVPN server

Next, in order to run an OpenVPN server, one needs to set up a certificate authority (CA) signs client certificates and authorizes them for login. In our case, we'll be using password authentication (for convenience) -- but OpenVPN still wants a CA setup and the server's certificate signed. Let's set up the CA for the OpenVPN server:

dnf install easy-rsa
cp -a /usr/share/easy-rsa/3 /root/openvpn-bridged-rsa
cd /root/openvpn-bridged-rsa
./easyrsa init-pki
./easyrsa build-ca
# Enter the CA password, then accept the defaults

Now we need to create and sign the certificate for the server (set the value of SERVER_ALIAS to an alias of your choice):

./easyrsa gen-dh
./easyrsa gen-req $SERVER_ALIAS nopass
./easyrsa sign-req server $SERVER_ALIAS
# Enter 'yes', then CA password

Finally, we copy the keys and certificates to a dedicated folder for OpenVPN:

mkdir /etc/openvpn/keys
chmod 700 /etc/openvpn/keys
cp pki/ca.crt pki/dh.pem "pki/issued/${SERVER_ALIAS}.crt" "pki/private/${SERVER_ALIAS}.key" /etc/openvpn/keys

We are now ready to configure OpenVPN. Set the variables based on your LAN's configuration (see ifconfig $ETH_IFACE output if unsure):


dnf install openvpn
firewall-cmd --permanent --add-service openvpn
firewall-cmd --reload

cat << EOF > /etc/openvpn/bridged.conf
port 1194
dev tap0
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/$SERVER_ALIAS.crt
key /etc/openvpn/keys/$SERVER_ALIAS.key # This file should be kept secret
dh /etc/openvpn/keys/dh.pem

# Password authentication
plugin /usr/lib64/openvpn/plugins/ openvpn

# Allow multiple client connections from the same user

# Client should attempt reconnection on link failure.
keepalive 10 120

# The server doesn't need root privileges
user openvpn
group openvpn

# Logging levels & prevent repeated messages
verb 4
mute 20
log-append /var/log/openvpn.log
status /var/log/openvpn-status.log

# Set some other options
push persist-key
push persist-tun

# Brings up tap0 since NetworkManager won't do it automatically (yet?)
script-security 2

OpenVPN will create the tap0 interface automatically when the OpenVPN server starts. NetworkManager is able to enslave the interface to the bridge, but won't bring tap0 online. For that, we install a simple script:

cat << EOF > /etc/openvpn/

# This should be done by NetworkManager... but it can't hurt.
/sbin/brctl addif \$br \$dev

# NetworkManager appears to be capable of enslaving tap0 to the bridge automatically, but won't bring up the interface.
/sbin/ifconfig \$dev promisc up
chmod +x /etc/openvpn/

Next, we need to create the PAM authentication configuration file for the OpenVPN password plugin:

cat << EOF > /etc/pam.d/openvpn
auth       substack     system-auth
auth       include      postlogin
auth       requisite user ingroup openvpn_pw quiet
account    required
account    include      system-auth
password   include      system-auth

This configuration file requires that the users logging in be a member of the openvpn_pw group. You can adjust the file as you see fit.

Finally open the OpenVPN port in the firewall and start the service:

firewall-cmd --permanent --add-service openvpn
firewall-cmd --reload
systemctl enable openvpn@bridged
systemctl start openvpn@bridged

Configure the firewall

By default, packets are filtered through iptables which can cause issues, as packets won't freely through between the interfaces. We can disable that behavior:

cat << EOF > /etc/modules-load.d/bridge.conf

cat << EOF > /etc/sysctl.d/bridge.conf
sysctl -p /etc/sysctl.d/bridge.conf

cat << EOF > /etc/udev/rules.d/99-bridge.rules
ACTION=="add", SUBSYSTEM=="module", KERNEL=="br_netfilter", RUN+="/sbin/sysctl -p /etc/sysctl.d/bridge.conf"

Note that I assume that net.ipv4.ip_forward=1 (having libvirt seems to configure this automatically). If not, you'll want to tune the sysctl parameter net.ipv4.ip_forward to a value of 1.

OpenVPN client configuration

That's it! Send a copy of /etc/openvpn/keys/ca.crt on the server to your clients, and you should now be able to connect to your OpenVPN server using this very simple client configuration (don't forget to replace your.server.fqdn with your server's IP address or FQDN):

dev tap
proto udp
remote your.server.fqdn 1194
resolv-retry infinite
ca ca.crt
verb 3
mute 20

Once connected, you should be able to ping any machine on the LAN as well as fire up Stream for a remote gaming session.

Appendix A: Steam on OS X

Small note, if you're running the Steam client on OS X there's a bug where the client only sends its UDP broadcast packets for in-home streaming discovery on the machine's primary (i.e. Ethernet or Wi-FI) interface. This nifty command captures those and re-broadcasts them over the VPN's interface (once again, substitute the value of BROADCAST_ADDR per your LAN settings):

sudo tshark -T fields -e data -l 'udp and dst port 27036' | script -q /dev/null xxd -r -p | socat - UDP-DATAGRAM:${BROADCAST_ADDR}:27036,broadcast

Special thanks to Larry Land for pointing that out in his blog post Run your own high-end cloud gaming service on EC2 (which is awesome and deserves a read, by the way).

The above command requires the Wireshark and socat utilities to be installed, which you can grab using homebrew:

brew install wireshark socat

and if you don't know your subnet's broadcast address, verify it with:

ifconfig tap0 | grep broadcast

Appendix B: Troubleshooting tips

Whenever possible, I like to use the most modern tooling available. This tends to bite me because documentation might not be as good or the feature set in the replacement tools might be lacking compared to the older tried and true tooling, but I try to always look forward. 'new' tooling like systemd, NetworkManager and firewalld might be rough around the edges, but I like modern feature set and consistency they bring. Most importantly, using them (instead of dropping various custom shell scripts here and there) feels a lot less like my server is held together with glue, which I like.

While trying different configurations, I discovered a few tricks or debugging commands that proved very useful to me - particularly while migrating commands from online resources intended for the older tooling to the tools mentioned above. Hopefully, you'll find them useful too!

It's always the firewall

The blame for most of the issues you will experience generally fall under firewall or routing issues.

First steps in testing should always be disabling the firewall (systemctl stop firewalld) and if that doesn't fix it, then move to checking the routes (route -n or netstat -rn).
If you've identified the firewall is to blame, re-enable it and identifying the root cause by adjusting your configuration while listening for packets to see when packets start flowing again.

Listening for packets

This will be your most used tool. If things don't go as expected, listen on each of the tap0 (client), tap0 (server) and br0 (server) interfaces and then generate some traffic to see how far the packets:


where PATTERN can select for hosts, ports or traffic type. In this case, a particular favorite of mine was icmp or udp port 27036 as this let me test by trying to ping a machine on the LAN from the VPN client, as well as see if the Steam UDP traffic was making it in/out.

Send UDP traffic

The iperf utility can be used to test if UDP traffic makes it to the OpenVPN server (run iperf -c server.ip -u -T 32 -t 3 -i 1 -p 27036) from the OpenVPN clients/LAN machines (run iperf -s -u -i 1 -p 27036).

Changing the zone of an interface

firewalld has different 'zones', each with different rules (see firewall-cmd --list-all-zones). Interfaces will have the rules from the default zone applied to them unless otherwise configured, which you can do as follows:

firewall-cmd --permanent --change-interface=IFNAME --zone=NEW_ZONE
firewall-cmd --reload

Adding raw IPTables rules to firewalld

e.g. to accept all packet forwarding on the bridge interface:

firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -i br0 -j ACCEPT

or to allow all packets flowing over br0 and tap0:

firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -i tap0 -j ACCEPT
firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -i br0 -j ACCEPT

Neither of these commands should be necessary given the netfilter sysctl parameters tweaked earlier, but certain OpenVPN options (such as client2client) change the packet flow and cause packets to flow over the interface, get filters, then re-injected which could cause them to suddenly be affected by the iptables rules.

Recall that firewall-cmd --reload needs to be called before the permanent rules will take effect.

Debugging IPTables

The default iptables -L listing isn't very helpful when inserting/deleting rules by their chain offset. The following command lists all rules, numerically, and displays line numbers:

iptables --line-numbers -L -n -v

You can also log dropped packets for further troubleshooting (here limited to 1/s, inserted at position 15 which was the position before the DROP rules on my machine):

iptables -I INPUT 15 -j LOG -m limit --limit 60/min --log-prefix "iptables dropped: " --log-level 4

Configuring a client VPN connection using only NetworkManager

nmcli c add type vpn ifname $VPN_IFNAME vpn-type openvpn
nmcli c modify $VPN_IFNAME
set dev = tap
set ca = /path/to/copy/of/ca.crt
set connection-type = password
set remote = your.server.fqdn
set comp-lzo = yes
set username = your_user
set connection-type = password
set vpn.secrets password = your_pw

'waiting for password' when connecting using Tunnelblock

When I was attempting to test my VPN connections using Tunnelblick on OS X, I experienced an annoying bug: When trying to connect, Tunnelblick would enter a 'Waiting for password' state, but never 'get' the password nor prompt for one. Log were misleading:

Tunnelblick: Obtained VPN username and password from the Keychain

No VPN password was stored in my keychain (verified using Keychain Fortunately, this post on the Sophos community forums correctly identified the issue as a bug after having copied/renamed a Tunnelblick connection.

Tunnelblick's preference file needs to be adjusted in order to correctly prompt for a password again:

# from
defaults delete net.tunnelblick.tunnelblick "${conname}-keychainHasPrivateKey"
defaults delete net.tunnelblick.tunnelblick "${conname}-keychainHasUsername"
defaults delete net.tunnelblick.tunnelblick "${conname}-keychainHasUsernameAndPassword"

Additional reading

Migrating a live server to another host with no downtime

I have had a 1U server co-located for some time now at iWeb Technologies' datacenter in Montreal. So far I've had no issues and it did a wonderful job hosting websites & a few other VMs, but because of my concern for its aging hardware I wanted to migrate away before disaster struck.

Modern VPS offerings are a steal in terms of they performance they offer for the price, and Linode's 4096 plan caught my eye at a nice sweet spot. Backed by powerful CPUs and SSD storage, their VPS is blazingly fast and the only downside is I would lose some RAM and HDD-backed storage compared to my 1U server. The bandwidth provided wit the Linode was also a nice bump up from my previous 10Mbps, 500GB/mo traffic limit.

When CentOS 7 was released I took the opportunity to immediately start modernizing my CentOS 5 configuration and test its configuration. I wanted to ensure full continuity for client-facing services - other than a nice speed boost, I wanted clients to take no manual action on their end to reconfigure their devices or domains.

I also wanted to ensure zero downtime. As the DNS A records are being migrated, I didn't want emails coming in to the wrong server (or clients checking a stale inboxes until they started seeing the new mailserver IP). I can easily configure Postfix to relay all incoming mail on the CentOS 5 server to the IP of the CentOS 7 one to avoid any loss of emails, but there's still the issue that some end users might connect to the old server and get served their old IMAP inbox for some time.

So first things first, after developing a prototype VM that offered the same service set I went about buying a small Linode for a month to test the configuration some of my existing user data from my CentOS 5 server. MySQL was sufficiently easy to migrate over and Dovecot was able to preserve all UUIDs, so my inbox continued to sync seamlessly. Apache complained a bit when importing my virtual host configurations due to the new 2.4 syntax, but nothing a few sed commands couldn't fix. So with full continuity out of the way, I had to develop a strategy to handle zero downtime.

With some foresight and DNS TTL adjustments, we can get near zero downtime assuming all resolvers comply with your TTL. Simply set your TTL to 300 (5 minutes) a day or so before the migration occurs and as your old TTL expires, resolvers will see the new TTL and will not cache the IP for as long. Even with a short TTL, that's still up to 5 minutes of downtime and clients often do bad things... The IP might still be cached (e.g. at the ISP, router, OS, or browser) for longer. Ultimately, I'm the one that ends up looking bad in that scenario even though I have done what I can on the server side and have no ability to fix the broken clients.

To work around this, I discovered an incredibly handy tool socat that can make magic happen. socat routes data between sockets, network connections, files, pipes, you name it. Installing it is as easy as: yum install socat

A quick script later and we can forward all connections from the old host to the new host:


# Stop services on this host
for SERVICE in dovecot postfix httpd mysqld;do
  /sbin/service $SERVICE stop

# Some cleanup
rm /var/lib/mysql/mysql.sock

# Map the new server's MySQL to localhost:3307
# Assumes capability for password-less (e.g. pubkey) login
ssh $NEWIP -L 3307:localhost:3306 &
socat unix-listen:/var/lib/mysql/mysql.sock,fork,reuseaddr,unlink-early,unlink-close,user=mysql,group=mysql,mode=777 TCP:localhost:3307 &

# Map ports from each service to the new host
for PORT in 110 995 143 993 25 465 587 80 3306;do
  echo "Starting socat on port $PORT..."
  socat TCP-LISTEN:$PORT,fork TCP:${NEWIP}:${PORT} &
  sleep 1

And just like that, every connection made to the old server is immediately forwarded to the new one. This includes the MySQL socket (which is automatically used instead of a TCP connection a host of 'localhost' is passed to MySQL).

Note how we establish a SSH tunnel mapping a connection to localhost:3306 on the new server to port 3307 on the old one instead of simply forwarding the connection and socket to the new server - this is done so that if you have users who are permitted on 'localhost' only, they can still connect (forwarding the connection will deny access due to a connection from a unauthorized remote host).

Update: a friend has pointed out this video to me, if you thought 0 downtime was bad enough... These guys move a live server 7km through public transport without losing power or network!

Alex Williamson's talk at KVM Forum 2014

Alex gave a very interesting talk at KVM Forum 2014 about the current state of VGA passthrough using KVM & VFIO:
Also, I think nVidia is making an incredibly silly choice (apparently accidentally) causing Code 43 in their drivers when virtualization is detected and refusing to fix the bugs. Virtualization is becoming evermore powerful and this is just going to push potential customers away to AMD. Once they establish a reputation for their cards not working well with virtualization, they're going to have trouble gaining custom confidence even if they reverse their stance on not fixing the Code 43 bugs.

Create a gaming virtual machine using VFIO PCI passthrough for KVM

This part of the Fedora 20 home server setup howtos will show you how to create a gaming KVM virtual machine by passing through real hardware using the new VFIO PCI passthrough technique. Your VM will achieve near real-world graphic and audio performance.