Cover photo by Grant Hussey @gthussey_art

I have a VPS where I host both public facing services (like this blog) and private services like node analytics, dashboards and other tools. In the past I've locked down these private services by simply adding basic http authentication. While this works, I would much rather completely lock down those routes and connect to them from the private network via VPN.

I set this up the other day and it turned out to be super simple using Traefik and Wireguard.

TLDR

Setup Wireguard VPN using this article, and create a middleware in Traefik that whitelists the VPNs public IP for specific services. Any services that use this middleware will only be accessible from your VPN!

  # Traefik container showing dynamic setup of middleware config
  - name: traefik
    image: traefik:v2.3
    ...
    labels:
      ...
      # private network ip whitelist
      traefik.http.middlewares.private-network-ipwhitelist.ipwhitelist.sourcerange: "127.0.0.1/32, <vpn-public-ip>/32"
      
  # Example of service locked down using middleware
  - name: airflow
    image: puckel/docker-airflow:1.10.9
    ...
    labels:
      ...
      traefik.http.routers.airflow.middlewares: private-network-ipwhitelist
  

Automating Wireguard Installation

I followed this article to install Wireguard, but converted the steps into the following Ansible role.

---
- name: Install wireguard
  apt:
    name: ["wireguard"]
    state: present
    update_cache: true

- name: Check that key files exist
  stat:
    path: /etc/wireguard/privatekey
  register: privatekey_result

- name: Generate keys
  shell: |
    wg genkey | sudo tee /etc/wireguard/privatekey | wg pubkey | sudo tee /etc/wireguard/publickey
  when: not privatekey_result.stat.exists

- name: Set permissions on wireguard privatekey
  file:
    path: /etc/wireguard/privatekey
    mode: "0600"

- name: Get wireguard private key
  slurp:
    src: "/etc/wireguard/privatekey"
  register: wg_privatekey

- name: Set host privatekey value
  set_fact:
    wg_privatekey: "{{ wg_privatekey['content'] | b64decode | replace('\n', '') }}"

- name: Add wireguard config file
  ansible.builtin.template:
    src: wg0.conf.j2
    dest: /etc/wireguard/wg0.conf
    owner: root
    mode: "0600"

- name: Stop service wireguard if already started
  ansible.builtin.systemd:
    name: wg-quick@wg0
    state: restarted
    enabled: yes

- name: Set ipv4 port forwarding
  sysctl:
    name: net.ipv4.ip_forward
    value: "1"
    state: present
    sysctl_set: yes

- name: Allow all access to port 53
  ufw:
    rule: allow
    port: "51820"
    proto: udp

- name: Add peers
  shell: |
    wg set wg0 peer {{ item.public_key }} allowed-ips {{ item.allowed_ips }}
  with_items: "{{ wg_peers }}"
roles/wireguard/tasks/main.yml

Which uses the following template for the Wireguard config. It's very standard, allowing internet access to the client through the server, running on the default port and using 10.0.0.1/24 subnet.

Note that the private key is comming in from Ansible.

[Interface]
Address = 10.0.0.1/24
SaveConfig = true
ListenPort = 51820
PrivateKey = {{ wg_privatekey }}
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o {{ ansible_default_ipv4.interface }} -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o {{ ansible_default_ipv4.interface }} -j MASQUERADE
roles/wireguard/templates/wg0.conf.j2

We now have to setup our client which is unfortunately manual. You need to create a new client key set using the app for your platform and edit the config to add your server as a peer. You will need the server's public key:

ssh <server-ip> "sudo cat /etc/wireguard/publickey"

Here is what mine looks like:

[Interface]
PrivateKey = <client-private>
Address = 10.0.0.2/24
DNS = 1.1.1.1, 1.0.0.1

[Peer]
PublicKey = <server-public>
AllowedIPs = 0.0.0.0/0
Endpoint = <server-ip>:51820
wireguard-client.conf

And now that you have a client private key, update your values file for the ansible role. Should look something like this:

wg_peers:
  - public_key: <client-public-key>
    allowed_ips: 10.0.0.2/32
host-values.yml

If you are confused or stuck anywhere backtrack and follow the guide I linked to above.

At this point you should be able to run this role against your server with the values file above, connect to your server via Wiregaurd and see that your public IP is now the same as your VPN.

Automating Traefik

Traefik has a really nifty feature where you can create a middleware that only allows a whitelists of IPs to access a route.

If you already have Traefik setup and running with your services, refer to the snippet at the top in the TLDR section. It's just two little lines!

Otherwise, here is the full docker container config from ansible dumped for any help with debugging. This is Traefik with all it's dynamic config and a container that is setup to only be accessible from VPN.

- name: traefik
    image: traefik:v2.3
    restart_policy: unless-stopped
    networks_cli_compatible: true
    networks:
      - name: main-net
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "traefik-letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    command:
      - "--log.level=DEBUG"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.network=main-net"
      - "--providers.docker.exposedByDefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.myresolver.acme.email=<redacted>"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.myresolver.acme.tlschallenge=true"
      - "--accesslog=true"
      - "--accesslog.filepath=/mnt/logs/access.log"
      - "--accesslog.bufferingsize=100"
    labels:
      traefik.enable: "true"
      traefik.docker.network: "main-net"
      traefik.http.routers.traefik.rule: Host(`traefik.{{ host }}`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.tls.certresolver: myresolver
      traefik.http.routers.traefik.middlewares: private-network-ipwhitelist
      # middleware redirect
      traefik.http.middlewares.redirect-to-https.redirectscheme.scheme: https
      # global redirect to https
      traefik.http.routers.redirs.rule: "hostregexp(`{host:.+}`)"
      traefik.http.routers.redirs.entrypoints: web
      traefik.http.routers.redirs.middlewares: redirect-to-https
      # private network ip whitelist
      traefik.http.middlewares.private-network-ipwhitelist.ipwhitelist.sourcerange: "127.0.0.1/32, 10.0.0.1/24, <vpn-public-ip>/32"
      
  - name: airflow
    image: puckel/docker-airflow:1.10.9
    become: true
    command: webserver
    user: root
    volumes:
      - /opt/airflow/dags:/usr/local/airflow/dags
      # rest are mounting to backup
      # everything in /toback will be backed up and restored when run
      - postgres:/toback/personal_server/docker/postgres
    networks:
      - name: main-net
    ports: []
    exposed_ports: []
    env:
      EXECUTOR: Local
      POSTGRES_DB: airflow
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: "{{ lookup('env', 'AIRFLOW_POSTGRES_PASSWORD') }}"
    labels:
      traefik.enable: "true"
      traefik.http.routers.airflow.rule: "Host(`airflow.{{ host }}`)"
      traefik.http.routers.airflow.entrypoints: "websecure"
      traefik.http.routers.airflow.tls.certresolver: "myresolver"
      traefik.http.services.airflow.loadbalancer.server.port: "8080"
      traefik.http.routers.airflow.middlewares: private-network-ipwhitelist