unifi

One of the neat and relatively undocumented feature of Unifi Security Gateway (USG) is the ability to specify alternate DNS servers sent with DHCP replies for specific clients, permitting you to do things like setup pihole for only a few specific devices on your LAN (e.g. the Smart TV or a streaming stick).

This is perfect as I didn't want to point my whole network at the pihole, as that would mean any technical issues with my pihole host (config errors, docker failing, etc) means the whole home Internet connection effectively going offline.

You can test out this feature interactively by SSHing into the USG and running these commands (replace capitals as appropriate):

configure
set service dhcp-server shared-network-name net_LANNAME_eth1_SUBNET-MASK subnet SUBNET/MASK static-mapping DASH-SEPARATED-MAC-ADDR ip-address LAN_STATIC_IP
set service dhcp-server shared-network-name net_LANNAME_eth1_SUBNET-MASK subnet SUBNET/MASK static-mapping DASH-SEPARATED-MAC-ADDR mac-address COLON:SEPARATED:MAC:ADDRESS
set service dhcp-server shared-network-name net_LANNAME_eth1_SUBNET-MASK subnet SUBNET/MASK static-mapping DASH-SEPARATED-MAC-ADDR static-mapping-parameters  "option domain-name-servers DNS_IP_FOR_OVERRIDE;"
commit
save
exit

This will edit the running configuration but rebooting or re-provisioning will lose these changes. To persist the configuration, create/edit your config.gateway.json with a snippet like this:

{
        "service": {
                "dhcp-server": {
                        "shared-network-name": {
                                "net_LANNAME_eth1_SUBNET-MASK": {
                                        "subnet": {
                                                "SUBNET/MASK": {
                                                        "static-mapping": {
                                                                "DASH-SEPARATED-MAC-ADDRESS": {
                                                                        "host-record": "disable",
                                                                        "ip-address": "LAN_STATIC_IP",
                                                                        "mac-address": "COLON:SEPARATED:MAC:ADDRESS",
                                                                        "static-mapping-parameters": "option domain-name-servers DNS_IP_FOR_OVERRIDE;"
                                                                }
                                                        }
                                                }
                                        }
                                }
                        }
                }
        }
}

Credit for discovering the syntax: tachyonforce on the Unifi forums

For those curious about the pihole setup specifically, I used docker-compose with the pihole/pihole image on a home server to get it running:

---
version: "2.2"

services:
  pihole:
    image: pihole/pihole
    container_name: pihole
    restart: unless-stopped
    environment:
      - TZ=America/Los_Angeles
      - ServerIP=192.168.1.15
      - WEBPASSWORD=arandompw
      - VIRTUAL_HOST=externalhostname
    ports:
      - "192.168.1.15:53:53/tcp"
      - "192.168.1.15:53:53/udp"
    volumes:
      - /srv/docker-vols/pihole/etc/pihole:/etc/pihole/
      - /srv/docker-vols/pihole/etc/dnsmasq.d:/etc/dnsmasq.d

Here I used the IP address of the server in the port mapping to as the server has multiple interfaces, and 53 is already used elsewhere. Specifying the IP ensures that Docker attempts to port map on the correct interface.

VIRTUAL_HOST is required because I use a reverse proxy to expose internal services, so the hostname must be provided to ensure dashboard URLs resolve correctly.

In my last post, I covered how to route packages from a specific VLAN through a VPN on the USG. Here, I will show how to use policy-based routing on Linux to route packets from specific processes or subnets through a VPN connection on a Linux host in your LAN instead. You could then point to this host as the next-hop for a VLAN on your USG to achieve the same effect as in my last post.

Note that this post will assume a modern tooling including firewalld and NetworkManager, and that subnet 192.168.10.0/24 is your LAN. This post will send packets coming from 192.168.20.0/24 to VPN, but you could customize that as you see fit (e.g. send specific only hosts from your normal LAN subnet instead).

VPN network interface setup

First, let's create a VPN firewalld zone so we can easily apply firewall rules just to the VPN connection:

firewall-cmd --permanent --new-zone=VPN
firewall-cmd --reload

Next, create the VPN interface with NetworkManager:

VPN_USER=openvpn_username
VPN_PASSWORD=openvpn_password

# Setup VPN connection with NetworkManager
dnf install -y NetworkManager-openvpn
nmcli c add type vpn ifname vpn con-name vpn vpn-type openvpn
nmcli c mod vpn connection.zone "VPN"
nmcli c mod vpn connection.autoconnect "yes"
nmcli c mod vpn ipv4.method "auto"
nmcli c mod vpn ipv6.method "auto"

# Ensure it is never set as default route, nor listen to its DNS settings
# (doing so would push the VPN DNS for all lookups)
nmcli c mod vpn ipv4.never-default "yes"
nmcli c mod vpn ipv4.ignore-auto-dns on
nmcli c mod vpn ipv6.never-default "yes"
nmcli c mod vpn ipv6.ignore-auto-dns on

# Set configuration options
nmcli c mod vpn vpn.data "comp-lzo = adaptive, ca = /etc/openvpn/keys/vpn-ca.crt, password-flags = 0, connection-type = password, remote = remote.vpnhost.tld, username = $VPN_USER, reneg-seconds = 0"

# Configure VPN secrets for passwordless start
cat << EOF >> /etc/NetworkManager/system-connections/vpn

[vpn-secrets]
password=$VPN_PASSWORD
EOF
systemctl restart NetworkManager

Configure routing table and policy-based routing

Normally, a host has a single routing table and therefore only 1 default gateway. Static routes can be configured for next-hops, this is configuring the system to route based a packet's destination address, and we want to know how route based on the source address of a packet. For this, we need multiple routing tables (one for normal traffic, another for VPN traffic) and Policy Based Routing (PBR) to define rules on how to select the right one.

First, let's create a second routing table for VPN connections:

cat << EOF >> /etc/iproute2/rt_tables
100 vpn
EOF

Next, setup an IP rule to select between routing tables for incoming packets based on their source addres:

# Replace this with your LAN interface
IFACE=eno1

# Route incoming packets on VPN subnet towards VPN interface
cat << EOF >> /etc/sysconfig/network-scripts/rule-$IFACE
from 192.168.20.0/24 table vpn
EOF

Now that we can properly select which routing table to use, we need to configure routes on the vpn routing table:

cat << EOF > /etc/sysconfig/network-scripts/route-$IFACE
# Always allow LAN connectivity
192.168.10.0/24 dev $IFACE scope link metric 98 table vpn
192.168.20.0/24 dev $IFACE scope link metric 99 table vpn

# Blackhole by default to avoid privacy leaks if VPN disconnects
blackhole 0.0.0.0/0 metric 100 table vpn
EOF

You'll note that nowhere do we actually define the default gateway - because we can't yet. VPN connections often dynamically allocate IPs, so we'll need to configure the default route for the VPN table to match that particular IP each time we start the VPN connection (we'll do so with a smaller metric figure than the blackhole above of 100, thereby avoiding the blackhole rule).

So, we will configure NetworkManager to trigger a script upon bringing up the VPN interface:

cat << EOF > /etc/NetworkManager/dispatcher.d/90-vpn
VPN_UUID="\$(nmcli con show vpn | grep uuid | tr -s ' ' | cut -d' ' -f2)"
INTERFACE="\$1"
ACTION="\$2"

if [ "\$CONNECTION_UUID" == "\$VPN_UUID" ];then
  /usr/local/bin/configure_vpn_routes "\$INTERFACE" "\$ACTION"
fi
EOF

In that script, we will read the IP address of the VPN interface and install it as the default route. When the VPN is deactivated, we'll do the opposite and cleanup the route we added:

cat << EOF > /usr/local/bin/configure_vpn_routes
#!/bin/bash
# Configures a secondary routing table for use with VPN interface

interface=\$1
action=\$2

tables=/etc/iproute2/rt_tables
vpn_table=vpn
zone="\$(nmcli -t --fields connection.zone c show vpn | cut -d':' -f2)"

clear_vpn_routes() {
  table=$1
  /sbin/ip route show via 192.168/16 table \$table | while read route;do
    /sbin/ip route delete \$route table \$table
  done
}

clear_vpn_rules() {
  keep=\$(ip rule show from 192.168/16)
  /sbin/ip rule show from 192.168/16 | while read line;do
    rule="\$(echo \$line | cut -d':' -f2-)"
    (echo "\$keep" | grep -q "\$rule") && continue
    /sbin/ip rule delete \$rule
  done
}

if [ "\$action" = "vpn-up" ];then
  ip="\$(/sbin/ip route get 8.8.8.8 oif \$interface | head -n 1 | cut -d' ' -f5)"

  # Modify default route
  clear_vpn_routes \$vpn_table
  /sbin/ip route add default via \$ip dev \$interface table \$vpn_table

elif [ "\$action" = "vpn-down" ];then
  # Remove VPN routes
  clear_vpn_routes \$vpn_table
fi
EOF
chmod 755 /usr/local/bin/configure_vpn_routes

Bring up the VPN interface:

nmcli c up vpn

That's all, enjoy!

Sending all packets from a user through the VPN

I find this technique particularly versatile as one can also easily force all traffic from a particular user through the VPN tunnel:

# Replace this with your LAN interface
IFACE=eno1

# Username (or UID) of user who's traffic to send over VPN
USERNAME=foo

# Send any marked packets using VPN routing table
cat << EOF >> /etc/sysconfig/network-scripts/rule-$IFACE
fwmark 0x50 table vpn
EOF

# Mark all packets originating from processes owned by this user
firewall-cmd --permanent --direct --add-rule ipv4 mangle OUTPUT 0 -m owner --uid-owner $USERNAME -j MARK --set-mark 0x50
# Enable masquerade on the VPN zone (enables IP forwarding between interfaces)
firewall-cmd --permanent --add-masquerade --zone=VPN

firewall-cmd --reload

Note 0x50 is arbitrary, as long as it the rule and firewall rule match, you're fine.

A little while back, I posted this on Reddit about setting up a Ubiquity Unifi Security Gateway (USG) or Edge Router Lite (ERL) to selectively route packets through a VPN interface; I wanted to elaborate a bit on the setup for this.

The goal

The goal was have my Unifi device establish two networks, one that behaves normally and another that routes all traffic through a VPN interface automatically. The value prop for a setup like this is that you can avoid having to configure each device & the VPN on each separately; simply connect to the network and that's it. It's simple, uses a single VPN connection for multiple devices and even lets friends & family use it easily with zero configuration.

In my case, I setup a second SSID and tied to different subnet (tagged VLAN) so simply by switching networks, you could gain VPN protection. Normally, this is very difficult to do because the router has a single default route; all packets not destined for local networks will exit using said default route.

This technique is made possible through the use of policy-based routing, which establishes multiple routing tables and rules on when to use a given table. This permits the router to determine the next-hop based on the source address, not the destination address.

Configuration

Here are some the basic steps to getting your USG configured:

# Setup route using table #1 with next-hop as VPN, blackhole if VPN is down
set protocols static table 1 route 0.0.0.0/0 blackhole distance 100
set protocols static table 1 interface-route 0.0.0.0/0 next-hop-interface vtun0 distance 2

# Set rules for when to send packets using routes from table 1
set firewall modify SOURCE_ROUTE rule 10 description "Traffic from VLAN 11 to VPN"
set firewall modify SOURCE_ROUTE rule 10 source address 192.168.20.0/24
set firewall modify SOURCE_ROUTE rule 10 modify table 1
set firewall modify SOURCE_ROUTE rule 10 action modify

# Apply the rule
set interfaces ethernet eth1 vif 11 firewall in modify SOURCE_ROUTE
commit
save

As long as the Ubiquity router is the default gateway (it should be if it's serving DHCP), machines on network 192.168.20.0/24 will now automatically route packets through the USG and then the USG will pick the vtun0 interface as its next-hop for those packets. Packets from other sources (e.g. your regular LAN) are routed normally through the WAN link.

Note that you can also set the next-hop to a host, by using this static route instead of the one above (note removal of the blackhole - you'll need to do that on the host specified in next-hop to avoid privacy leaks):

# Setup route table #2 with next-hop as VPN via local server
set protocols static table 1 route 0.0.0.0/0 next-hop 192.168.20.100

This is useful if you have a home server connected to VPN, and want to route packets through its VPN connection instead of the USG (some additional setup required; more on that in this post).

Troubleshooting

If this setup does not work as expected, the easiest way to troubleshoot is to verify connectivity. tcpdump will be your best friend here. Something like tcpdump -i interface -A port 80 will trace all HTTP traffic on the interface supplied, and

You'll want to verify connectivity in this order:

1. Verify source packets leave a host

Use any machine on the 192.168.20.0/24 network to test and start generating some traffic, ensuring the packets do hit the network. You can use something simple like curl google.com to trigger some traffic, and monitor the network interface with tcpdump per above to make sure the packets are sent out.

2. Verify packets are reaching your next-hop

Your next-hop is probably the USG/ERL router, but could also be an IP on your network as well. Now that we've confirmed packets are leaving, make sure they are arriving by inspecting the LAN-side interface on your configured next-hop.

Connect over SSH to the next hop (if it's a USG/ERL read this) and run sudo -i. Use ip a to list all interfaces & configured IPs; this should let you pick out the interface name associated to an IP on the 192.168.20.0/24 network.

Now run tcpdump against that interface and then generate some traffic on the test host from step 1. Do you see them arriving?

3. Verify packets are exiting your next-hop on the VPN interface

If you've made it this far packets arrive on your next-hop so let's make sure it's forwarding out through the right interface. First, list all configured policy-based routing rules with ip rule - you should expect to see 0 (default table) and 1 (for certain marked packets). List out each routing table using ip rule show table X to make sure things look as you'd expect. For example, ip route show table 1:

default dev vtun0  scope link
blackhole default  metric 100

Or if you configured next-hop as a host instead:

default via 192.168.20.100 dev eth1.11 

Now run tcpdump on the shown interface and verify if packets are existing as expected.

Note for Unifi users

Lastly, note that if you use a USG instead of a ERL, these settings will not be persisted. Your settings will be overwritten by Unifi Controller after any provision or reboot operation -- you will need to manually persist them by exporting to a config.gateway.json file.

I just picked up some new networking gear, so this will be the first of a multi-part blog post about my learnings configuring Unifi gear.

One issue I noticed right away was that it is not possible, via CLI nor GUI, to configure fixed IP address for a host that relies on more than 1 of the configured networks/VLANs. Since I have a home server (user VLAN) that is also hosting the controller softare (management VLAN) and also acts as a gateway for sending packets over its VPN interface (VPN VLAN), this was necessary for me.

It is possible but requires a bit of manual configuration using a config.gateway.json file. First, if you have configured a fixed IP for the host, unset it.

Then, merge in the DHCP mappings in your config.gateway.json file:

{
  "service":{
    "dhcp-server":{
      "shared-network-name":{
        "LAN_192.168.1.0-24":{
          "subnet":{
            "192.168.1.0/24":{
              "static-mapping":{
                "00-aa-22-bb-44-cc.mgmt":{
                  "ip-address":"192.168.1.5",
                  "mac-address":"00:aa:22:bb:44:cc"
                }
              }
            }
          }
        },
        "LAN_Users_192.168.10.0-24":{
          "subnet":{
            "192.168.10.0/24":{
              "static-mapping":{
                "00-aa-22-bb-44-cc.users":{
                  "ip-address":"192.168.10.5",
                  "mac-address":"00:aa:22:bb:44:cc"
                }
              }
            }
          }
        },
        "LAN_VPN_192.168.20.0-24":{
          "subnet":{
            "192.168.20.0/24":{
              "static-mapping":{
                "00-aa-22-bb-44-cc.vpn":{
                  "ip-address":"192.168.20.5",
                  "mac-address":"00:aa:22:bb:44:cc"
                }
              }
            }
          }
        }
      }
    }
  }
}

The key here is that the string child of the static-mapping node must be unique. Unifi will put in the MAC separated by dashes by default, so above I just tacked on the VLAN name to each name.

Re-provision your USG and you should be good to go. If you run into trouble an want to debug DHCP req/ack sequences, setup verbose logging:

configure
set service dhcp-server global-parameters 'log-facility local2;'
set system syslog file dhcpd facility local2 level debug
set system syslog file dhcpd archive files 5
set system syslog file dhcpd archive size 5000
commit

You'll find the DHCP log under /var/log/user/dhcpd. Simply reboot to go back to normal logging.