I wanted a homelab that is actually secured, not just "secured enough for a weekend project". The goal was a sensible, defensible baseline of security and network isolation that I would be comfortable running long-term, that I could grow into, and that would not embarrass me from inside.

This is a hobby project that run it on my own time - but it is built to the standard I would defend and experience at work (all the time). It is where I host the side projects I ship after hours, and I keep it on operating system (or even hardware) I fully own. The network is carved up into distinct octets so concerns stay isolated - a subnet per purpose - and so that adding capacity later is a data change rather than a redesign / architecture revolution.

There is also one, trivial goal mixed in with the fun: I wanted a VPN with a stable, static public IP physically located in Sweden. I tried several large, well-known commercial VPNs specifically to test whether they present cleanly as a Swedish endpoint, and too often they gave themselves away as "a VPN" rather than a "resident connection" - possibly because they offer a rotating IP from a shared pool that many sites already recognise (poor reputation address pool). I never tested one of their static-IP tiers, so that may be exactly the difference. Either way, running my own hub on a fixed Swedish IP lets me combine the useful with the enjoyable. (Whether that actually makes me look more like "a Swede" - <digression>a little wink to Bettie here 😉</digression> - than a rotating premium endpoint is a nuance I will come back to honestly at the end - the short version is: it helps, but it is not magic of course).

Why this is worth the effort

When you have spent enough years shipping software, a homelab (not even at your "home" physically) like this stops being a novelty and becomes leverage you fully control (and fun). Day to day I work on managed platforms and I love them. Here I own "every" abstraction layer (every = from kernel space every machine upwards, sometimes hardware and the physical network devices as well - in cases when fraction of machines stays at my home). Precisely, it earns its keep by giving me:

• a place to run real infrastructure code (Ansible, Terraform and Kubernetes manifests) against real machines, with real network segmentation and real security consequences, rather than a throwaway sandbox;
• a private build-and-host environment for the side projects I work on in my spare time - source hosting, CI, code-quality gates, artifact and object storage, a database (sql, non-sql, documents, cache), ingress, secrets, observability, SRE;
• a place to keep the operational disciplines in working order: backups, key rotation, disaster recovery, least-privilege access, supply-chain and vulnerability scanning/patching - the practices that are expensive to rehearse in production and cheap to keep fluent at home;
• and, put simply, the standards that keep this system coherent are the same standards to which everyone holds their professional work.

You can reasonably think of the end state as a small private cloud. It is an overstatement - I am not rebuilding a "hyperscaler in a cupboard" - but it is the right direction of travel, and holding the design to that bar is what keeps the decisions honest.

Originally the whole thing lived inside Google Cloud. It worked, but a managed platform hides the layers I most want ownership of: the network edge, the firewall, the SSH surface, the boot-time state, kernel elements, low-level packages. For a system I answer for personally, I would rather own bare Linux end to end and control every layer than delegate them to a control plane I cannot inspect. So I moved to a model where I own the machines and configure them from a bare OS upward, with the entire configuration captured in code.

Everything below is driven by an idempotent Ansible repository. The control machine is my laptop (macOS, commands run in zsh); the managed hosts are plain Linux (one of the popular distro, the LTS branch) - VPS instances. The whole thing is layered: bootstrap an admin user, harden SSH, apply common system settings, configure the firewall, add fail2ban, then lay down the WireGuard mesh, and finally verify all the inventory.

What I have today

The core are three roles of machine, each in its own purpose-named inventory group:

• vpn1 - a WireGuard VPN hub, always on. This is my day-to-day operational control path.
• vpn2 - a second WireGuard hub, kept as a cold standby: normally powered off and commented out of the inventory, brought up for a monthly test or when I actually need it. Critically, it is a different provider in a different data centre and different country from vpn1.
• bastion1 - the SSH jump host. It is the only machine I SSH into directly over the VPN; everything else I reach by jumping through it.

The security baseline applied to every host is:

• root SSH login disabled;
• SSH password authentication disabled (public-key only);
• UFW active, default-deny inbound, default-allow outbound;
• fail2ban watching sshd;
• unattended upgrades and the APT daily timers enabled;
• no public SSH anywhere - 22/tcp is only ever allowed on a WireGuard interface, from a specific mesh address;
• monitoring agent.

The public IP of every host refuses SSH outright. The only way in is over WireGuard, and then only via the bastion.

The universal baseline

One principle keeps the fleet coherent as it grows: every host receives the same hardened baseline before it ever does anything specialised. Any Postgres cluster, a Kubernetes node, Kafka node, Redis server, a mail server - none of them is exempt. The machine's purpose is a layer added on top. The baseline is non-negotiable and identical everywhere. That is what stops a fleet from degenerating into a collection of hand-tuned pets, each secured slightly differently and audited never.

Precisely, playbooks/site.yml applies these roles to hosts: all first, before any per-purpose play runs:

• common - timezone, locale, a small set of baseline packages and configuration for zsh, vim, the hostname (set once and pinned so "vendor-init" cannot silently reset it), and unattended upgrades with the APT daily timers. Security patches land without me having to remember to log in (like in corporate, real-work-world).
• users - my personal administrative account, its membership of sudo, passwordless sudo written to /etc/sudoers.d/<username> and validated with visudo before it is committed, and the YubiKey-backed SSH public keys (I'll extend this in Chapter 1 below, please stay tuned).
• ssh - the hardening drop-in (root login off, passwords off, pubkey only), validated with sshd -t before it is allowed to take effect.
• firewall - UFW, default-deny inbound, and the WireGuard-only SSH rules derived from the mesh dictionary. No host is ever reachable on a public port it does not explicitly need.
• fail2ban - the sshd jail, so even the WireGuard-fronted SSH surface has brute-force protection behind it;
• node_exporter + alloy - every host emits metrics and ships logs from the moment it joins the fleet, not only the ones I remember to instrument. Observability belongs in the baseline, not bolted on later.

The philosophy throughout is validate before you apply, and prefer a drop-in to editing a primary config file - sshd -t for SSH, visudo -cf for sudoers, and drop-ins under sshd_config.d/ and cloud.cfg.d/ rather than rewriting the vendor's files. A change that would break a service is caught before it reaches the running daemon, not after.

What I would add to the baseline next

The baseline is deliberately lean today, but there are a handful of roles I consider natural additions once a machine starts carrying real workloads:

• auditd - a kernel-level audit trail of privileged actions and file access, which is the sort of thing you want to have already been recording when you go looking for it;
• a CIS-style hardening pass - kernel sysctl tightening, sensible mount options, and disabling unused services, ideally checked by a tool such as Lynis so the hardening is measured rather than assumed.
• reboot_required reporting - randomly, during the "night", detect /var/run/reboot-required after unattended upgrades and surface it, so kernel updates do not sit unapplied indefinitely.
• chrony / time sync - accurate time is a quiet prerequisite for sane logs, TLS validation and any future certificate work.
• a backup agent (restic or borg to the MinIO/object store) - so a host is recoverable by policy, not by luck, as soon as it holds state worth keeping.

I add these as their own roles, each with a dedicated tag and a matching assertion in verify.yml, so the baseline and its verification always grow together. A feature that is configured but not verified is a feature I do not actually trust, I believe. 

Chapter 1: Start with the YubiKeys

If you take one thing from this post: begin with hardware-backed SSH keys. Everything else in the design assumes my SSH identity is anchored in hardware, so this is genuinely step zero.

My admin SSH keys are all sk-ssh-ed25519@openssh.com keys - FIDO/U2F keys whose secret half lives on a YubiKey, never on the laptop's disk. Generating one on macOS is a single command (I create them as resident credentials so the key handle can be pulled back off the key itself later - see the trade-offs below):

ssh-keygen -t ed25519-sk -O resident -O application=ssh:homelab \
  -C "<username>@yubikey-1" -f ~/.ssh/id_ed25519_sk_yubikey1

The -t ed25519-sk picks the FIDO key type; -O resident stores the credential on the key so a fresh laptop can recover it; -O verify-required can be added to force a PIN on every use. The private half never exists on disk - only a small non-secret key handle does.

I have four YubiKeys, and I treat them the way you would treat any irreplaceable secret:

• one on me, day to day;
• one at home, as the immediate spare;
• two more in two independent locations, each at least several tens of kilometres away from where I live, in secure and monitored premises.

All four private keys are pre-enrolled with my SSH keys, so any one of them can get me in. They are protected by PINs and a passphrase. I do not persist SSH sessions in an agent: the socket cache lives for three minutes, after which it evaporates and I have to re-authenticate. It is mildly annoying and completely worth it.

The public halves of all four keys are committed to the repo (public keys are safe to commit) in inventory/production/files/authorized_keys, one key per line, and pushed to every host by the users role. The role loads them as a list and adds each one additively (state: present, no exclusive), so enrolling a fifth key is just another line in that file plus a re-run:

- name: Ensure admin user exists
  ansible.builtin.user:
    name: "{{ admin_user }}"
    shell: "{{ admin_user_shell }}"
    groups: "{{ admin_user_groups }}"
    append: true
    create_home: true
    state: present

- name: Ensure SSH public keys are configured for admin user
  ansible.posix.authorized_key:
    user: "{{ admin_user }}"
    key: "{{ item }}"
    state: present
  loop: "{{ admin_user_ssh_public_keys }}"

The list itself comes straight from that committed file, via a lookup in group_vars/all.yml:

admin_user_ssh_public_keys: "{{ lookup('file', inventory_dir + '/files/authorized_keys').splitlines() }}"

So authorised access is version-controlled and reproducible. The file is the policy. verify.yml later re-reads the on-host authorized_keys and asserts every committed key is actually present (more on that in Chapter 3).

I also keep a written, step-by-step recovery runbook for the day the laptop dies, so that recovery is a checklist and not an improvisation.

Advantages

• The secret never touches disk. A stolen, infected or cloned laptop disk yields no SSH secret.
• Physical possession plus a PIN/passphrase is required - genuine "two-factor", in hardware.
• The short agent cache means a walk-away laptop is not an open door for long.
• Four keys across four locations means no single fire, theft or loss locks me out.

Disadvantages

• Operational friction. Re-touching and re-authenticating every three minutes is real. That is the price, and I pay it deliberately. Maybe I should extend this some day.
• Key handles. OpenSSH FIDO keys also need a small non-secret key handle (the .pub side). Unless you create the keys as resident credentials (ssh-keygen -t ed25519-sk -O resident), you must back that handle up - lose it and you have to re-derive it. Resident keys let you pull the handle back off the YubiKey itself with ssh-keygen -K, which is the recovery-friendly choice.
• Enrolment discipline. Four keys means four enrolments to keep in sync, a new key means visiting your authorized_keys and re-running Ansible.

What happens if I lose the key or a laptop?

If I lose (or destroy) one YubiKey. Nothing breaks. Any of the other three gets me in. I revoke/replace the lost one at leisure. Because there is a PIN and a passphrase, a found or stolen key is not immediately useful to whoever has it.

If my laptop dies (disk fire, theft, hardware failure) but the keys survive. This is the case my recovery runbook is written for. My SSH identity survived - it was always on the YubiKeys, never on the laptop. The thing I actually lost was the laptop's WireGuard private keys (more on that in the next chapter). Recovery will be a simple process - on a replacement Mac, pull the resident key handles back off a YubiKey, reinstall WireGuard, re-import the backed-up tunnel profiles, and confirm a jump into the bastion prompts me to touch a YubiKey:

ssh-keygen -K

# Sanity-check the fingerprint the key now offers:
ssh-keygen -l -f ~/.ssh/id_ed25519_sk_yubikey1.pub

If the jump succeeds, I am fully back: identity and network path both restored, and - because the laptop's public key never changed - nothing on the servers or in the repo needs touching.

Chapter 2: The WireGuard mesh - two hubs, two providers

The network side is a full access mesh driven by a single source-of-truth data structure. There is no hand-maintained peer list anywhere; two dictionaries (wg_hubs and wg_members) in group_vars/all.yml describe the whole topology, and the Ansible roles, firewall rules and verification checks are all derived from that data (public keys and IP shown as placeholders): 

wg_hubs:
  vpn1:
    subnet: 10.8.0.0/24
    hub_address: 10.8.0.1
    listen_port: 51820
    endpoint: "<vpn1-public-ip>:51820"
    hub_pub_key: "<vpn1-hub-public-key>"

  vpn2:
    subnet: 10.9.0.0/24
    hub_address: 10.9.0.1
    listen_port: 51820
    endpoint: "<vpn2-public-ip>:51820"
    hub_pub_key: "<vpn2-hub-public-key>"

wg_members:
  vpn1:
    addresses: { vpn1: 10.8.0.1, vpn2: 10.9.0.4 }
    spoke_pub_keys: { vpn2: "<vpn1-wg1-public-key>" }

  vpn2:
    addresses: { vpn1: 10.8.0.4, vpn2: 10.9.0.1 }
    spoke_pub_keys: { vpn1: "<vpn2-wg1-public-key>" }

  bastion1:
    addresses: { vpn1: 10.8.0.3, vpn2: 10.9.0.3 }
    spoke_pub_keys:
      vpn1: "<bastion1-wg0-public-key>"
      vpn2: "<bastion1-wg1-public-key>"

  laptop:
    addresses: { vpn1: 10.8.0.2, vpn2: 10.9.0.2 }
    spoke_pub_keys:
      vpn1: "<laptop-vpn1-profile-public-key>"
      vpn2: "<laptop-vpn2-profile-public-key>"

That is it. Everything else - hub peer lists, spoke tunnels, firewall rules, verification - works based on these two dictionaries. A key I do not have yet is written as a literal placeholder, and every role treats a placeholder as "skip, do not fail" (I will come back to why that matters).

The topology and the roles

Hubs (vpn1, vpn2). Each hub owns a subnet - vpn1 is 10.8.0.0/24, vpn2 is 10.9.0.0/24 (the different-octet split I mentioned at the start). Each hub is a WireGuard server on its own tap interface (always named wg0), enables IPv4 forwarding, and masquerades its own subnet so spokes get routing.

Every host is a spoke of every other hub. This is the whole point. bastion1 has a tunnel to both hubs (wg0 faces vpn1, wg1 faces vpn2). Each hub is also a spoke of the other hub, over a single extra wg1 tunnel. So the mesh has no single point of network failure. Losing one hub's WireGuard config does not lost access to the rest of the fleet, because the surviving hub still routes to everyone.

The bastion is the only front door. No host allows public SSH. Every non-bastion host accepts 22/tcp only on its WireGuard interface, only from the bastion's address in that subnet. The bastion itself accepts SSH only from my laptop's WireGuard address. So the path is always laptop -> (WireGuard) -> bastion1 -> (ssh -J) -> target. Interface naming is fixed and boring on purpose: wg0 always faces vpn1, wg1 always faces vpn2. A hub is never a spoke of itself, so a hub's one extra spoke tunnel always lands on wg1 (its own wg0 is reserved for its hub server). This convention is enforced by an assertion in the wireguard_client role, so a bad data entry fails loudly instead of quietly clobbering a hub's own key:

- name: Verify no spoke tunnel on a hub host resolves to wg0 (RF6)
  ansible.builtin.assert:
    that:
      - item.interface != 'wg0'
    fail_msg: >-
      Hub host {{ inventory_hostname }} must never run a spoke tunnel on wg0 -
      that interface is reserved for its own hub server.
  loop: "{{ wireguard_client_tunnels }}"
  when: "'vpn' in group_names"

SSH hardening

Before any of the network work, every host is hardened. I write a drop-in rather than editing the primary sshd_config, and I validate it with sshd -t before it is allowed to land - a broken config never reaches a running daemon:

- name: Configure SSH hardening
  ansible.builtin.copy:
    dest: "{{ ssh_hardening_config_file }}"
    content: |
      PermitRootLogin {{ ssh_permit_root_login }}
      PasswordAuthentication {{ ssh_password_authentication }}
      KbdInteractiveAuthentication {{ ssh_kbd_interactive_authentication }}
      ChallengeResponseAuthentication {{ ssh_challenge_response_authentication }}
      PubkeyAuthentication {{ ssh_pubkey_authentication }}
      PermitEmptyPasswords {{ ssh_permit_empty_passwords }}
    validate: /usr/sbin/sshd -t -f %s
  notify: Restart ssh

One subtlety worth the war story: the file is called 00-ansible-hardening.conf, not 99-.... The provider image ships its own drop-in named 00-<provider_name>-auth.conf, and sshd honours the first occurrence of each directive in lexical order.

A 99-* file would sort after the provider's and silently lose. Naming mine 00-ansible-* makes it sort first and win. Ugly, but works.

The bastion is the only front door

No host allows public SSH. The rule that enforces it is not written by hand per host - it is derived from the mesh dictionary. For each hub, I emit one UFW rule allowing 22/tcp on the interface that reaches that hub, from the bastion's address in that hub's subnet:

firewall_interface_rules: >-
  {%- set ns = namespace(rules=[]) -%}
  {%- for hub in wg_hubs.keys() -%}
    {%- set ns.rules = ns.rules + [{
      'rule': 'allow', 'port': 22, 'proto': 'tcp',
      'interface': wg_ssh_interface_for_hub[hub],
      'direction': 'in',
      'from_ip': wg_members.bastion1.addresses[hub],
    }] -%}
  {%- endfor -%}
  {{ ns.rules }}

firewall_removed_tcp_ports:
  - 22

(Jinja has no list comprehension, hence the namespace() accumulator pattern.) The bastion itself is the one exception - group_vars/bastion.yml overrides the same structure to source SSH from the laptop's address instead of its own, because it is the jump host. The path is therefore always laptop -> (WireGuard) -> bastion1 -> (ssh -J) -> target:

# Day-to-day reach vpn1 by jumping through the bastion over WireGuard:
ssh -J <username>@10.8.0.3 <username>@10.8.0.1

# The public IP, by contrast, simply refuses:
ssh <username>@<vpn1-public-ip>

Why two hubs at two providers?

Redundancy that survives a provider outage, not just a machine outage. If one datacenter has a bad day, the other is on an entirely separate network with a separate blast radius. vpn2 being a cold standby keeps the running cost near zero while still giving me a proven, tested fallback - I bring it up monthly, verify the whole mesh "goes green", then power it back off.

The two providers are also not configured identically. vpn1 gives direct root SSH to bootstrap. The cloud provider hosting vpn2 does not hand out root over SSH - it ships an unprivileged account and expects me to be granted sudo through its console first. It also gates inbound traffic at a VPC firewall that sits "in front of" the host's own UFW. WireGuard would not even handshake until I opened udp:51820 at that cloud layer, entirely separately from UFW. So on that host, SSH lockdown is a two-layer story - UFW on the box, and the provider's firewall in front of it. Opening the WireGuard UDP port to the world there is fine, incidentally - WireGuard is silent to unauthenticated peers.

How a hub renders its peers

The hub role (roles/wireguard) does not read a hand-written peer list. It walks wg_members, includes every member that has a real key for this hub, and skips placeholders with a friendly message instead of failing:

- name: Collect hub peers from the mesh dictionary
  ansible.builtin.set_fact:
    wireguard_hub_peers: "{{ wireguard_hub_peers + [wireguard_hub_peer_candidate] }}"
  vars:
    wireguard_hub_peer_candidate:
      name: "{{ wireguard_hub_peer_display_names[item.key] | default(item.key) }}"
      public_key: "{{ item.value.spoke_pub_keys[inventory_hostname] }}"
      allowed_ips: "{{ item.value.addresses[inventory_hostname] }}/32"
      sort_key: "{{ item.value.addresses[inventory_hostname].split('.') | map('int') | list }}"
  loop: "{{ wg_members | dict2items }}"
  when:
    - item.key != inventory_hostname
    - item.value.addresses[inventory_hostname] is defined
    - (item.value.spoke_pub_keys[inventory_hostname] | default('')) | length > 0
    - not (item.value.spoke_pub_keys[inventory_hostname] | default('')).startswith('REPLACE_')

(startswith('REPLACE_') - if I don' have a key yet - "cold start" for a new host).

The peers are then sorted by address (parsing the dotted quad into integers, so 10.8.0.10 sorts after 10.8.0.2, not before it) before templating. That sort is not cosmetic - it keeps the rendered config byte-identical across runs, so adding a new member later does not reshuffle existing lines and needlessly fire the "restart WireGuard" handler on an otherwise-unchanged hub. The template itself is tiny:

{% raw %}[Interface]
Address = {{ wireguard_address }}
ListenPort = {{ wireguard_listen_port }}
PrivateKey = {{ wireguard_server_private_key }}

{% for peer in wireguard_hub_peers_sorted %}
[Peer]
# {{ peer.name }}
PublicKey = {{ peer.public_key }}
AllowedIPs = {{ peer.allowed_ips }}

{% endfor %}{% endraw %}

How a spoke derives its tunnels

The spoke role (roles/wireguard_client) runs on every host and derives one tunnel per other hub, picking the fixed interface and marking the tunnel active only if the hub's key is real:

- name: Derive one spoke tunnel per other hub in the mesh
  ansible.builtin.set_fact:
    wireguard_client_tunnels: "{{ wireguard_client_tunnels + [wireguard_client_tunnel_entry] }}"
  vars:
    wireguard_client_tunnel_entry:
      hub: "{{ item.key }}"
      interface: "{{ 'wg1' if inventory_hostname in wg_hubs else wg_hub_interfaces[item.key] }}"
      address: "{{ wg_members[inventory_hostname].addresses[item.key] }}/32"
      hub_pub_key: "{{ item.value.hub_pub_key }}"
      endpoint: "{{ item.value.endpoint }}"
      allowed_ips: "{{ item.value.subnet }}"
      active: >-
        {{ (item.value.hub_pub_key | default('')) | length > 0
           and not (item.value.hub_pub_key | default('')).startswith('REPLACE_') }}
  loop: "{{ wg_hubs | dict2items }}"
  when: item.key != inventory_hostname

Each tunnel is split-tunnel (AllowedIPs is just that hub's subnet) with PersistentKeepalive = 25 so it stays reachable through NAT, and each notifies a per-interface handler - so a change to wg1 can never restart wg0 and cut my live control path.

Advantages

• No single point of failure at the network layer. Two hubs, two providers, full mesh. If vpn1 disappears, I switch my laptop to the vpn2 profile and still reach the bastion.
• No public attack surface for SSH. The public IPs simply refuse 22/tcp. Port scanners find nothing to talk to.
• Single source of truth. Adding a new host to the mesh is one dictionary entry. The roles, firewall rules and verification all pick it up automatically - no code changes.
• Safe partial states. A not-yet-provisioned peer is represented by a placeholder key (REPLACE_). The roles simply skip it rather than failing. The mesh stays "green" while a new host is half-built, and starts being fully verified the moment its real key lands.
• Idempotent and auditable. A second run reports no changes. Everything is in git - public keys only, private keys are generated on each host and never leave it.

Disadvantages

• The laptop's two VPN profiles are mutually exclusive. Each profile only sees its own hub's subnet, and they cannot both be active at once. In practice this means one Ansible run cannot reach both subnets at the same time - I run against one hub's hosts on one profile, and the other hub on the other. Fine today - if the fleet grows across both hubs, I will likely drive everything through the bastion (the one host that sees both subnets).
• Chicken-egg problem on lockdown. Because SSH is only reachable over WireGuard, I can lock myself out mid-run if I apply the firewall before the tunnel and jump path are proven. The whole procedure is therefore carefully ordered - bring the tunnel up, confirm the ssh -J path works from a separate terminal, and only THEN apply the firewall - bastion last, because it carries everything.
• A hub is only "done" after its firewall role runs. That role is what installs the NAT/masquerade rule. Before it runs, a freshly bootstrapped hub has forwarding enabled but no NAT, so spokes get a tunnel with no internet. Diagnosable, but the first time it's tricky.
• Provider asymmetry is cognitive load. Root vs no-root, UFW vs an extra VPC firewall - I have to hold both models in my docs up to date.

What happens if I lose a hub or bastion?

• One hub's config gets broken (e.g. WireGuard config). No big deal. The other hub still routes. I switch my laptop to the other profile, reach the bastion, jump to the broken hub, and fix it by re-run Ansible, which is the source of truth for the config.
• A whole hub machine vanishes from the network (dead or unreachable). Same story from the access point of view - the surviving hub keeps me in. If it is the cold-standby vpn2, I have lost nothing that was live. If it is vpn1, I bring vpn2 online (a documented, phased, simple procedure), and rebuild vpn1 from scratch - the entire host configuration lives in the repo, so abreprovision is a bootstrap plus a playbook run, not complicated archaeology.
• The bastion breaks. The bastion is the one machine on the live path that I treat with the most care, precisely because it carries everything - which is why it is always the last host I touch during any firewall change, and why I never disconnect a working session until a fresh one is proven from a second terminal. If it does break, the provider's rescue console is the fallback, and the repo rebuilds it.

Chapter 3: A verify.yml playbook

Configuring a host is only half the job. The other half is proving it is still configured the way I think it is. playbooks/verify.yml runs against every host, changes nothing, and asserts the whole baseline: root login off, passwords off, pubkey on, UFW active, fail2ban active, the unattended-upgrades timer enabled, the hostname correct, and - crucially - that there is no global SSH rule anywhere and that every committed admin key is actually present on the box.

The pattern throughout is check, then assert - a read-only command that registers its output (changed_when: false), followed by an assert on it:

- name: Check effective SSH configuration
  ansible.builtin.command: /usr/sbin/sshd -T
  register: verify_sshd_config
  changed_when: false

- name: Verify root SSH login is disabled
  ansible.builtin.assert:
    that:
      - "'permitrootlogin no' in verify_sshd_config.stdout"
    fail_msg: "Root SSH login is not disabled."

The check I value most is the negative one - proving the thing that must not exist really does not:

- name: Verify global SSH access is absent
  ansible.builtin.assert:
    that:
      - >-
        verify_ufw_ssh.stdout_lines
        | select('match', '^22/tcp\s')
        | reject('search', 'wg0')
        | reject('search', 'wg1')
        | list | length == 0
    fail_msg: "UFW still has a global 22/tcp rule; SSH must only be reachable over WireGuard."

And because the keys are hardware-backed and multiple, verify.yml reads the on-host authorized_keys back and asserts each committed public key is there:

- name: Read admin user authorized_keys file
  ansible.builtin.slurp:
    src: "/home/{{ admin_user }}/.ssh/authorized_keys"
  register: verify_authorized_keys_raw
  changed_when: false

- name: Verify all admin SSH public keys are present
  ansible.builtin.assert:
    that:
      - item in (verify_authorized_keys_raw.content | b64decode)
  loop: "{{ admin_user_ssh_public_keys }}"

The good point is that verify.yml is driven by the same mesh dictionary as the roles. Early in the play it derives per-host fact lists, tagging each hub and peer with two flags - key_real (the key is not a placeholder) and in_inventory (the hub is currently present) - and only hard-asserts when both are true. Otherwise it emits a soft warning:

- name: Verify SSH is allowed on each verified interface from the expected source
  ansible.builtin.assert:
    that:
      - >-
        verify_ufw_ssh.stdout_lines
        | select('search', '22/tcp on ' + item.interface)
        | select('search', item.source)
        | select('search', 'ALLOW')
        | list | length > 0
  loop: "{{ verify_wg_interfaces }}"
  when: item.verified | bool

That single when: is what lets the whole suite stay green with the cold-standby vpn2 commented out of inventory, and start hard-asserting vpn2 automatically the moment its keys are real and it is uncommented - with no change to the playbook. The expected result of a run is failed=0, unreachable=0, and ideally changed=0 from site.yml too, which is my "idempotency canary".

As I wrote, the verification and the configuration evolve from one source of truth, so they cannot silently drift apart. Disadvantage: some assertions are coupled to how ufw status renders its columns - a phrasing change in a future UFW could make a select('search', ...) filter need adjusting.

Chapter 4: Adding another host

This is the pay-off of pushing everything through wg_hubs/wg_members. Onboarding a new machine into the mesh is, in the normal case, a configuration/data change. The workflow:

1. Add the host to the inventory (host_vars/<host>.yml with common_hostname, a line in hosts.ini under the right group).

2. Add a wg_members entry (plus a wg_hubs entry if it is a new hub), using REPLACE_* placeholders for keys I do not have yet.

3. Bootstrap and baseline it. With the keys still placeholders, the tunnels are gracefully-skipped - but the local keypairs are still generated, so I can read the public halves back out:

ansible-playbook playbooks/bootstrap.yml -l <host> -e ansible_user=root
ansible-playbook playbooks/site.yml -l <host>

ansible <host> -m command -a 'cat /etc/wireguard/publickey-wg0' --become
ansible <host> -m command -a 'cat /etc/wireguard/publickey-wg1' --become

4. Paste those real public keys over the REPLACE_* placeholders, then re-apply the two WireGuard roles so the new peer/tunnels light up across the mesh:

ansible-playbook playbooks/site.yml --tags wireguard,wireguard_client

5. Confirm the tunnel and the ssh -J path work from a separate terminal, and only then lock the firewall down - always with --limit, never a blind full run, so:

ansible-playbook playbooks/site.yml --tags firewall --limit <host>

6. ansible-playbook playbooks/verify.yml - which, as above, now hard-asserts the new host automatically.

The REPLACE_* convention is what makes step 3 safe: a half-built host is a solid, valid state of the data, not an error. The mesh stays green throughout the rollout, and I never have to comment code in and out to add a box.

The trade-off is that this elegance rests on fairly dense Jinja - the namespace() accumulators, the dict2items loops, the placeholder guards. It is more to read than a hardcoded config, and it asks a reader to trust that "it is all derived" before they have followed every filter. That is a deliberate bet - for a fleet that will only grow, the data-driven model pays for its complexity many times over. For three or five static hosts it would be premature. I am building for the trajectory, not the current "host count" - which is the call I would make on any system I expect to outlive its first design.

Chapter 5: Shipping a real app

The whole point of the secure baseline is that real workloads can now sit on top of it and the result behaves, loosely, like a private cloud. Let me make that by walking through adding a simple PostgreSQL database and a FastAPI service, with security testing baked in and a deployment onto Kubernetes - because a homelab that cannot ship an app is just a lab.

Step 1: PostgreSQL as a managed building block

I run Postgres on the cluster (or on a dedicated mesh host, if I want it off the critical path), and I do not hand-write its password anywhere. Vault mints short-lived database credentials on demand. A minimal Kubernetes StatefulSet for PostgreSQL, with the password sourced from a Secret that Vault populates:

apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: postgres
spec:
 serviceName: postgres
 replicas: 1
 selector:
   matchLabels: { app: postgres }
 template:
   metadata:
     labels: { app: postgres }
   spec:
     containers:
       - name: postgres
         image: postgres:18
         ports:
           - containerPort: 5432
         env:
           - name: POSTGRES_DB
             value: appdb
           - name: POSTGRES_USER
             value: app
           - name: POSTGRES_PASSWORD
             valueFrom:
               secretKeyRef:
                 name: postgres-credentials
                 key: password
         volumeMounts:
           - name: data
             mountPath: /var/lib/postgresql/data
 volumeClaimTemplates:
   - metadata: { name: data }
     spec:
       accessModes: ["ReadWriteOnce"]
       resources:
         requests: { storage: 10Gi }

Note the discipline is the same one from the mesh - secrets never live in the repo. The manifest references a Secret (comes from Vault).

Step 2: A FastAPI service

The app itself is deliberately small - a health endpoint plus one route that talks to PostgreSQL:

import os
import asyncpg
from fastapi import FastAPI

app = FastAPI()
DB_DSN = os.environ["DB_DSN"]

@app.get("/health")
async def health():
  return {"status": "ok"}

@app.get("/widgets/{widget_id}")
async def get_widget(widget_id: int):
  connection = await asyncpg.connect(DB_DSN)
  try:
    row = await connection.fetchrow(
      "SELECT id, name FROM widgets WHERE id = $1", widget_id
    )
       
    if row:
      return dict(row) 
               
    return {"status": "error", "details": "Object not found in database"}

  finally:
    await connection.close()

Containerised with a small, pinned base image (pinning matters for the scanning step below):

FROM python:3.14-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ ./app/
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Step 3: GitLab pipeline

This is the part I propose to not skip, because the whole "private cloud" framing only earns the name if the supply chain is watched. My GitLab pipeline runs, on every push, in roughly this order:

• Dependency / outdated-package checks - flag stale and vulnerable Python dependencies before they ship.
• Vulnerability scanning of the built image - scan the container for known CVEs in OS packages and libraries, fail the build on high severity.
• Static analysis - SonarQube quality gate plus a Python linter.
• Deploy to Kubernetes - only if every gate above is green.

A trimmed .gitlab-ci.yml file that captures the shape of it:

stages: [test, security, deploy]

dependency-audit:
  stage: security
  script:
    - pip install pip-audit
    - pip-audit -r requirements.txt
    - pip list --outdated

sast:
  stage: security
  script:
    - pip install bandit
    - bandit -r app/
    - sonar-scanner

image-scan:
  stage: security
  script:
    - docker build -t "$IMAGE" .
    - trivy image --exit-code 1 --severity HIGH,CRITICAL "$IMAGE"

deploy:
  stage: deploy
  needs: [dependency-audit, sast, image-scan]
  script:
    - kubectl apply -f k8s/postgres.yaml
    - kubectl apply -f k8s/fastapi.yaml
    - kubectl rollout status deployment/fastapi
  environment: homelab

The tools are interchangeable (pip-audit/safety, bandit, trivy/grype, SonarQube) - the principle is the point: outdated-dependency detection, vulnerability scanning, and static analysis are all gates, and deployment only happens when they pass. It is the posture I would hold a production platform to... There is no reason a personal one should be held to less.

Step 4: Expose and watch

nginx (or an ingress controller) fronts the service - because the whole cluster lives inside the WireGuard mesh, I can keep it entirely private, or expose a single vetted route through the Swedish hub if I want it reachable from a Swedish IP. Prometheus scrapes /metrics, Loki collects the logs via alloy, and Grafana shows me both - so when something misbehaves at 3am I have the same observability I would expect from myself at work.

Advantages of the "private cloud" framing

• End-to-end ownership. Source, CI, scanning, secrets, runtime, storage, ingress and monitoring are all mine, on hardware I control (some hosts are "under my desk" physically), inside a network with no public attack surface.
• Realistic practice. The workflow mirrors a production platform closely enough that the skills transfer directly to the day job.
• Cost and privacy. No per-request billing, tiers, no third party in the data path, and side projects can incubate privately for as long as I like.

Disadvantages - the honest caveats

• I am now the platform team. Every upgrade, CVE, certificate rotation and backup is on me. A managed cloud absorbs enormous operational toil that I am choosing to take back.
• "Private cloud" is an overstatement. I have no multi-region control plane, no managed autoscaling, and my "high availability" is two hubs and a standby - fine for a hobby, not a space flight control system.
• Blast radius of my own mistakes. The same ownership that teaches me the most is the thing that can take everything down at once if I am careless - which is why the recovery chapters below exist.

Chapter 6: Backing up the WireGuard keys

Here is the trap the whole design creates. Since SSH is reachable only over WireGuard, and WireGuard needs the laptop's private key, a dead laptop gives you this loop:

• to SSH in - I need a WireGuard
• to have WireGuard - I need the laptop's private key
• the private key - was only on the dead laptop

The YubiKeys save my SSH identity, but they do nothing for the WireGuard private key, which lives only on the laptop (it's a SPOF). So I keep a one-time offline backup of both tunnel profiles (the full .conf for each, private key included) on the same encrypted USB sticks, in the same secure locations, as the YubiKeys. Four copies, multiple locations - the same redundancy model.

A laptop profile is an ordinary WireGuard .conf - the same shape the wireguard_client role templates for the servers, just full-tunnel:

[Interface]
Address = 10.8.0.2/32
PrivateKey = <laptop-vpn1-private-key>

[Peer]
PublicKey = <vpn1-hub-public-key>
Endpoint = <vpn1-public-ip>:51820
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25

Everything else in that file is already public and in the repo (the address, the hub public key, the endpoint) - only the private key is irreplaceable, and that is exactly what the backup protects. When I take the backup I verify grabbed the right one by re-deriving the public key and matching it against the repo:

grep PrivateKey ~/wg-backup/vpn1.conf | awk '{print $3}' | wg pubkey

(must equal wg_members.laptop.spoke_pub_keys.vpn1 in group_vars/all.yml)

The backup is static - the laptop's keys do not rotate on their own - so I only refresh it if I ever regenerate a laptop key or a hub is re-keyed.

The hard rule: those .conf files never go into the repository. The repo holds public keys only. Private keys stay on the encrypted sticks, of course.

With that backup in place, "laptop died" is a fifteen-minute checklist: restore the SSH handle from a YubiKey, reinstall WireGuard, re-import the two .conf files, activate one profile, jump into the bastion. Without it, you are reduced to the deliberately painful last resort - an out-of-band provider console shell on a hub, generating a fresh key by hand and editing the mesh dictionary - which is precisely the situation the offline backup exists to avoid.

Epilogue

The theme running through all of this is: make the recovery story explicit for every resource before you need it. For each thing I could lose - a YubiKey, the laptop, a hub's config, a whole hub machine, the bastion - I know exactly what breaks, what survives, and what the recovery steps are, and most of them are boring precisely because I designed the redundancy up front (and make a fire drills, and wrote it down).

It is more rigour than a hobby project is often given - and that is precisely the point. The friction is the feature: hardware-anchored identity, no public SSH, a two-provider mesh with no single point of failure, everything in idempotent code, and a break-glass runbook for the bad day. None of it is exotic - it is simply the standard I would apply to anything I am responsible for. That is the foundation. On top of something I actually trust, I get to build the interesting part - a private-cloud-shaped platform for my after-hours projects, with PostgreSQL cluster, FastAPI, Kubernetes, Kafka, security-gated CI/CD, monitoring and a secret manager.

Appendix: does the Swedish hub really make me look 🇸🇪 Swedish?

I want to come back to this for a minute. Running my own WireGuard hub on a fixed, residential-or-datacentre public IP in Sweden genuinely helps compared with a big commercial VPN, but it is not a magic hat, and it is worth being precise about "why":

• The IP is stable and unshared. Commercial VPNs hand out addresses from pools that thousands of people cycle through. Those ranges are widely catalogued and frequently flagged. My hub's single, unchanging IP has none of that reputation baggage - it behaves like one ordinary connection, because it is one.
• But the IP's classification still matters. What actually gives a commercial VPN away is usually that its address sits in an ASN/range publicly labelled as "hosting" or "VPN/proxy". If my Swedish hub is on a data-centre IP, a determined site can still see it is a hosting provider, not a home broadband line - so I would look like "a server in Sweden", not necessarily "a Swedish resident on a sofa". A residential/consumer static IP in Sweden would look far more like a local person than any commercial VPN can.
• The rest of the fingerprint has to agree. Being "seen as Swedish" is more than the IP, of course - timezone, browser locale/language headers, DNS resolver location, WebRTC leaks all get inspected, OS configuration, other devices inside a network... Because my hub can be a genuine full tunnel with DNS pinned to the Swedish side, I can make all of those line up - which is often where the commercial VPNs quietly fail (e.g. a mismatched DNS exit or a leaked local IP).

So: yes, materially more convincing than a rotating premium endpoint, chiefly because it is a dedicated, low reputation, static IP with a consistent fingerprint - and the gap would widen further with a residential IP rather than a data-centre one. It is emphatically not a tool for defeating anyone's terms of service. It is about being legitimately and consistently present in a market I actually use. And it is a lovely example of the theme of this whole post - when you own the infrastructure end to end, the boring details (which IP, which DNS, which region) are yours to get right.