A little while back, I posted this on Reddit about setting up a Ubiquity Unifi Security Gateway (USG) or Edge Router Lite (ERL) to selectively route packets through a VPN interface; I wanted to elaborate a bit on the setup for this.

The goal

The goal was have my Unifi device establish two networks, one that behaves normally and another that routes all traffic through a VPN interface automatically. The value prop for a setup like this is that you can avoid having to configure each device & the VPN on each separately; simply connect to the network and that’s it. It’s simple, uses a single VPN connection for multiple devices and even lets friends & family use it easily with zero configuration.

In my case, I setup a second SSID and tied to different subnet (tagged VLAN) so simply by switching networks, you could gain VPN protection. Normally, this is very difficult to do because the router has a single default route; all packets not destined for local networks will exit using said default route.

This technique is made possible through the use of policy-based routing, which establishes multiple routing tables and rules on when to use a given table. This permits the router to determine the next-hop based on the source address, not the destination address.

Configuration

Here are some the basic steps to getting your USG configured:

# Setup route using table #1 with next-hop as VPN, blackhole if VPN is down
set protocols static table 1 route 0.0.0.0/0 blackhole distance 100
set protocols static table 1 interface-route 0.0.0.0/0 next-hop-interface vtun0 distance 2

# Set rules for when to send packets using routes from table 1
set firewall modify SOURCE_ROUTE rule 10 description "Traffic from VLAN 11 to VPN"
set firewall modify SOURCE_ROUTE rule 10 source address 192.168.20.0/24
set firewall modify SOURCE_ROUTE rule 10 modify table 1
set firewall modify SOURCE_ROUTE rule 10 action modify

# Apply the rule
set interfaces ethernet eth1 vif 11 firewall in modify SOURCE_ROUTE
commit
save

As long as the Ubiquity router is the default gateway (it should be if it’s serving DHCP), machines on network 192.168.20.0/24 will now automatically route packets through the USG and then the USG will pick the vtun0 interface as its next-hop for those packets. Packets from other sources (e.g. your regular LAN) are routed normally through the WAN link.

Note that you can also set the next-hop to a host, by using this static route instead of the one above (note removal of the blackhole - you’ll need to do that on the host specified in next-hop to avoid privacy leaks):

# Setup route table #2 with next-hop as VPN via local server
set protocols static table 1 route 0.0.0.0/0 next-hop 192.168.20.100

This is useful if you have a home server connected to VPN, and want to route packets through its VPN connection instead of the USG (some additional setup required; more on that in this post).

Troubleshooting

If this setup does not work as expected, the easiest way to troubleshoot is to verify connectivity. tcpdump will be your best friend here. Something like tcpdump -i interface -A port 80 will trace all HTTP traffic on the interface supplied, and

You’ll want to verify connectivity in this order:

1. Verify source packets leave a host

Use any machine on the 192.168.20.0/24 network to test and start generating some traffic, ensuring the packets do hit the network. You can use something simple like curl google.com to trigger some traffic, and monitor the network interface with tcpdump per above to make sure the packets are sent out.

2. Verify packets are reaching your next-hop

Your next-hop is probably the USG/ERL router, but could also be an IP on your network as well. Now that we’ve confirmed packets are leaving, make sure they are arriving by inspecting the LAN-side interface on your configured next-hop.

Connect over SSH to the next hop (if it’s a USG/ERL read this) and run sudo -i. Use ip a to list all interfaces & configured IPs; this should let you pick out the interface name associated to an IP on the 192.168.20.0/24 network.

Now run tcpdump against that interface and then generate some traffic on the test host from step 1. Do you see them arriving?

3. Verify packets are exiting your next-hop on the VPN interface

If you’ve made it this far packets arrive on your next-hop so let’s make sure it’s forwarding out through the right interface. First, list all configured policy-based routing rules with ip rule - you should expect to see 0 (default table) and 1 (for certain marked packets). List out each routing table using ip rule show table X to make sure things look as you’d expect. For example, ip route show table 1:

default dev vtun0  scope link
blackhole default  metric 100

Or if you configured next-hop as a host instead:

default via 192.168.20.100 dev eth1.11 

Now run tcpdump on the shown interface and verify if packets are existing as expected.

Note for Unifi users

Lastly, note that if you use a USG instead of a ERL, these settings will not be persisted. Your settings will be overwritten by Unifi Controller after any provision or reboot operation — you will need to manually persist them by exporting to a config.gateway.json file.