How to Build a Fault-Tolerant Internal DNS Layer For a Homelab

When the number of service IP addresses in my home toolkit started growing, it became worth thinking about DNS. When accessing my services, I want to use a nice, human-friendly address in the browser, available only over an active VPN connection, backed by a fault-tolerant DNS layer, and secured with well-known Let's Encrypt TLS certificates without browser warnings.

A quick note on names in this post before we'll begin: my real internal domain is a private one that I would rather not publish here, so throughout this article I use example.com (and int.example.com) as a stand-in. Wherever you see monitoring1.int.example.com, just mentally substitute your own domain. Public IP addresses of my VPS hosts are omitted on purpose. The private 10.x.x.x addresses shown are RFC 1918 and harmless.

The problem I actually wanted to solve

My homelab runs on a small fleet of VPS hosts joined into a WireGuard access mesh. Over time the fleet grew: two VPN hubs, a bastion jump host, two monitoring servers running Prometheus, Loki and Grafana, two more hosts earmarked for DNS, Kubernetes cluster, GitLab... Every one of those hosts lives at a private WireGuard address, in example: 10.8.0.5 (let's assume that is my monitoring1 host with Observability stack).

That works, but it does not scale well. I don't want to remember that Grafana on monitoring1 is 10.8.0.5 while monitoring2 (alternative observability stack) works under 10.8.0.6, and that the bastion is 10.8.0.3... What I wanted was simple and boring:

Connect the VPN (my WireGuard).
Type https://monitoring1.int.example.com:3000 in the browser.
Get a valid padlock, no certificate warning.
Public sites like https://google.com keep working exactly as before (and based on my internal DNS cache).

The hard constraint underneath the convenience is the interesting part. DNS is a convenience layer, not a root of trust. If my internal DNS breaks completely, I must still be able to reach every host by its raw WireGuard internal IP and repair things. Pretty names are for daily comfort, IP addresses are for disaster recovery, and the recovery path must have as few dependencies as possible. The whole design below falls out of taking that one sentence seriously. 🙂

The starting point: two bare, hardened hosts

I had two machines destined to become DNS servers, called dns1 and dns2, did not start as DNS servers yet. They started as bare, clean Linux-based hosts that had already been through the standard fleet baseline:

administrative user created, with my YubiKey-backed SSH public keys installed (hardware-backed sk-ssh-ed25519@openssh.com keys);
root SSH login disabled, password authentication disabled, public-key only;
SSH reachable only over WireGuard, only via the bastion (no public SSH);
UFW active, default-deny inbound, everything closed except what is explicitly opened;
fail2ban running, unattended upgrades and the APT daily timers enabled;
WireGuard spoke tunnels to both hubs, monitoring agents shipping metrics and logs.

If you want to read more about it - here you can.

So this project was purely additive - put an internal DNS resolver on top of an already-hardened baseline, without weakening any of it.

A deliberate architectural choice up front: dns1 and dns2 are treated as two independent, equal instances. Not a primary/secondary pair, no zone transfers, no shared runtime state, no VIP, no load balancer in front. Each one is a complete resolver on its own. The client is simply told about both addresses and fails over between them. The reason is failure isolation, of course. I did not want to introduce a new coordinating component whose own failure could take down two otherwise healthy resolvers at once.

Why Unbound?

I only need a tiny, tightly-scoped set of private records, essentially: hostname -> WireGuard IP.

For that, standing up BIND, PowerDNS or a database-backed DNS platform would be absurd overkill. Unbound is primarily a recursive, caching resolver, but it can also serve a small amount of local data via local-zone and local-data. That is exactly the shape of the problem:

internal names under int.example.com are answered locally from a handful of local-data records;
everything else (for example google.com) is forwarded to trusted public resolvers and cached locally.

No dynamic DNS, no GUI, no zone transfers, no API, no application-driven registration, no database. The configuration management repository stays the single source of truth, and the DNS daemons are just runtime consumers of it.

The consequence I care about - losing dns1 or dns2 loses no configuration. A replacement is rebuilt from the repo, applied by Ansible, is authoritative, neither DNS host is.

The single source of truth: one dictionary, one loop

The whole fleet's WireGuard mesh is already described by a single dictionary in the Ansible inventory. Every member has an address in each hub subnet. The key insight for DNS is that I already have, in one place, the exact hostname -> WireGuard IP mapping I want to serve. I refuse to maintain that mapping a second time in a hand-written zone file, because then the inevitable happens:

ansible inventory says one IP;
DNS configuration says another IP;
nobody notices until something breaks.

So the Unbound records are generated from the same mesh dictionary the rest of the fleet already uses. In practice that means a small Jinja loop in the role that walks every mesh member and emits one A record per host, pointing at that host's address in the always-on hub's subnet (the vpn1 subnet, 10.8.0.0/24, because that hub is always up, vpn2 is only a breaking-glass access after disaster).

The general idea is simple:

{%- for member_name, member_data in wg_members.items() -%}
  {%- if member_name != 'laptop' -%}
    local-data: "{{ member_name }}.int.example.com. IN A {{ member_data.addresses.vpn1 }}"
  {%- endif -%}
{%- endfor -%}

Add a host to the mesh dictionary and its DNS record appears automatically on the next apply (except my localhost-laptop). There is no second list to keep in sync.

Unbound configuration assumptions

Ansible writes a single drop-in file, /etc/unbound/unbound.conf.d/00-ansible-int-example.conf, rather than touching the package's primary unbound.conf. The file is validated with unbound-checkconf before it is written, so a malformed render fails fast instead of taking the resolver down.

The rendered result, on dns1, looks like this (addresses are this host's own):

server:
  # Bind only to WireGuard addresses plus loopback, never 0.0.0.0.
  interface: 10.8.0.7
  interface: 10.9.0.7
  interface: 127.0.0.1
  ip-freebind: yes
  port: 53
  do-ip4: yes
  do-udp: yes
  do-tcp: yes
  do-ip6: no
 
  hide-identity: yes
  hide-version: yes
  harden-glue: yes
  harden-dnssec-stripped: yes
  qname-minimisation: yes
 
  # Who may query: every mesh member, in both hub subnets. Default deny:
  access-control: 0.0.0.0/0 refuse
  access-control: 127.0.0.0/8 allow
  access-control: 10.8.0.2/32 allow
  access-control: 10.8.0.3/32 allow
  # ... one line per mesh member per subnet...

  # Private namespace served locally. 'A' records only:
  local-zone: "int.example.com." static
  local-data: "bastion1.int.example.com. IN A 10.8.0.3"
  local-data: "monitoring1.int.example.com. IN A 10.8.0.5"
  local-data: "monitoring2.int.example.com. IN A 10.8.0.6"
   # ... and so on...

# Everything else goes to three trusted public resolvers:
forward-zone:
  name: "."
  forward-first: no
  forward-addr: 1.1.1.1
  forward-addr: 1.0.0.1
  forward-addr: 8.8.8.8
  forward-addr: 8.8.4.4
  forward-addr: 9.9.9.9
  forward-addr: 149.112.112.112

Binding to specific addresses, never `0.0.0.0`

Unbound listens on this host's two WireGuard addresses and on 127.0.0.1, and nowhere else. It is never an open resolver on a public interface. Listening on the second hub address as well (the 10.9.0.x one) is a deliberate detail I will come back to in the failure analysis.

Parameter: `ip-freebind: yes`

The WireGuard addresses only exist once the tunnels are up. On a reboot, unbound.service can start before wg-quick@wgN has finished bringing the interface up, and a plain bind() to a not-yet-present address fails with EADDRNOTAVAIL, leaving the resolver dead. ip-freebind sets the Linux IP_FREEBIND socket option so Unbound can bind before the address is live. It changes nothing about which addresses it serves. It only relaxes the boot ordering.

Layered access control

There are three independent gates in front of this resolver, and it takes all three failing to expose it:

It binds only to WireGuard addresses (private interfaces).
UFW opens 53/udp and 53/tcp on the WireGuard interfaces only, and only from the specific mesh-member source addresses.
Unbound's own access-control refuses everything by default and allows only the known mesh members. Unbound matches access-control by most-specific prefix, so the 0.0.0.0/0 refuse line does not override the /32 allows - it is the "catch-all floor".

Parameter: `forward-first: no`

If all forwarders are unreachable, Unbound does not silently fall back to iterating from the root servers. It fails the public query. Internal names still resolve, because they are answered locally. This keeps outbound DNS strictly limited to the configured forwarders and makes the failure behaviour predictable.

Multiple upstream providers, RTT-based failover

I list 6 public DNS server addresses:

2x Cloudflare (1.1.1.1 + 1.0.0.1)
2x Google (8.8.8.8 + 8.8.4.4)
2x Quad9 (9.9.9.9 + 149.112.112.112)

Unbound tracks round-trip time per forwarder, prefers the fastest (and healthy) one, and fails over automatically when one stops answering. Three independent providers means the public-DNS path stays up unless three separate anycast networks fail at once.

More about `systemd-resolved`

Most Linux-based systems ships systemd-resolved listening on 127.0.0.53. I deliberately do not touch it. Unbound serves mesh clients on the WireGuard addresses and on 127.0.0.1 - the host's own local name resolution stays on systemd-resolved at 127.0.0.53. It's a different loopback addresses, so there is no port conflict, and the host keeps resolving names for wget, curl and others - completely independently of the resolver it serves to the mesh.

Configuration - step by step

The role is applied by a dedicated play that targets only the DNS group. In order, on each host:

Install unbound (an ordinary APT package) and dnsutils. The latter is installed so the verification playbook has dig available without depending on anything else.
Ensure the drop-in directory exists (just for the sake of sanity).
Render the drop-in config from the mesh dictionary and validate it with unbound-checkconf before writing.
Enable and start the unbound service.

The whole role is deliberately small. Here is the task file in full, lightly anonymised:

- name: Ensure unbound and dnsutils are installed
  ansible.builtin.apt:
    name:
      - unbound
      - dnsutils
    state: present
    update_cache: true

- name: Ensure /etc/unbound/unbound.conf.d directory exists
  ansible.builtin.file:
    path: /etc/unbound/unbound.conf.d
    state: directory
    owner: root
    group: root
    mode: "0755"

# unbound-checkconf validates the file before it is written, so a bad
# render fails fast instead of being deployed and taking the resolver down:
- name: Configure Unbound (int.example.com local-zone + public forwarders)
  ansible.builtin.template:
    src: 00-ansible-int-example.conf.j2
    dest: /etc/unbound/unbound.conf.d/00-ansible-int-example.conf
    owner: root
    group: root
    mode: "0644"
    validate: "unbound-checkconf %s"
  notify: Restart unbound

- name: Ensure unbound service is enabled and started
  ansible.builtin.systemd:
    name: unbound
    enabled: true
    state: started

That is genuinely all of it. Two things are worth pointing out. dnsutils (which provides dig) is installed here on purpose, not assumed to be present, so that the verification playbook's resolution checks are self-contained and do not depend on some other role having pulled it in first. And the config is written to a drop-in under unbound.conf.d/ rather than editing the package's primary unbound.conf, as I wrote, so a distribution upgrade that rewrites the main file never evaporates my configuration by mistake.

The handlers/main.yml that the template task notifies is a single, boring handler:

- name: Restart unbound
  ansible.builtin.systemd:
    name: unbound
    state: restarted

The interesting part of the role is not the tasks, it is the template that the second task renders. That template is where the mesh dictionary is turned into local-data, access-control and interface lines. The listing shown earlier in "Unbound configuration assumptions" is the rendered output of this template on dns1, so the loops that produce it (for example the local-data) live in the role's defaults and template, so adding a host to the mesh dictionary is the only change ever needed to give it a DNS record.

Separately, the firewall role opens 53/udp and 53/tcp on the WireGuard interfaces for exactly the mesh-member source addresses. Because both the DNS records and the firewall sources are derived from the same dictionary, they can never drift apart. The rule generation itself is a small Jinja loop that walks every hub and every mesh member and emits an allow rule per protocol:

firewall_dns_interface_rules: >-
  {%- set ns = namespace(rules=[]) -%}
  {%- for hub in wg_hubs.keys() -%}
    {%- for name, m in wg_members.items() -%}
      {%- for proto in ['udp', 'tcp'] -%}
        {%- set ns.rules = ns.rules + [{
          'rule': 'allow',
          'port': dns_port,
          'proto': proto,
          'interface': wg_ssh_interface_for_hub[hub],
          'direction': 'in',
          'from_ip': m.addresses[hub],
        }] -%}
      {%- endfor -%}
    {%- endfor -%}
  {%- endfor -%}
  {{ ns.rules }}

The exact same wg_members dictionary drives the DNS records, the firewall sources and Unbound's own access-control list. There is one source of truth and three consumers of it.

The verification playbook then asserts, on the DNS hosts, shortly:

the unbound service is active;
it is bound to the expected WireGuard addresses on both UDP and TCP, and not to a wildcard address;
an internal name resolves to the expected address;
a public name resolves.

Once it is live, confirming it by hand is trivial. From either DNS host:

dig @127.0.0.1 monitoring1.int.example.com +short
# ... should be 10.8.0.5

dig @127.0.0.1 google.com +short
# ... should be a public address

And from the laptop, after pointing the VPN profile's DNS at the two resolvers (DNS = 10.8.0.7, 10.8.0.8 in the WireGuard client config):

dig monitoring1.int.example.com +short
# ... should be 10.8.0.5

ssh bastion1.int.example.com
# ... should work, using a name instead of an IP

The client is told about both resolvers directly. There is no load balancer deciding which one to use. The operating system's resolver simply has two nameservers and moves on to the second if the first does not answer.

Potential failure analysis: what happens when things break?

I will walk through the cases from mild to severe.

One DNS server fails (let's say `dns1`)

dns2 keeps serving both internal and public names, from its own local records and its own cache and forwarders. The client had both 10.8.0.7 and 10.8.0.8 configured as nameservers, so it fails over to dns2. Normal work continues. I verified this as well - with unbound stopped on dns1, a query sent directly to dns2 still answered correctly, while a query to dns1 timed out as expected.

# on dns1
systemctl stop unbound

# from the client, still fine because dns2 answers
dig @10.8.0.8 bastion1.int.example.com +short

10.8.0.3

# and dns1 is genuinely down, not cached
dig @10.8.0.7 bastion1.int.example.com +short

... (timeout)

One honest caveat about the client side of this failover. dig talks to a nameserver directly and is a clean way to prove the servers behave, but it bypasses the operating system's own resolver stack. Real applications do not. On macOS in particular there are several caching layers stacked in front of the configured nameservers: mDNSResponder (the system-wide DNS client and Unicast DNS cache, which also handles multicast DNS and service discovery) sits at the bottom, and above it individual applications, notably the browsers, keep their own in-process DNS caches. macOS also does not do naive round-robin between the two configured nameservers. mDNSResponder tracks per-server behaviour and will keep sending queries to the first nameserver until it decides that server is unhealthy, and it applies its own timeouts and negative-caching before it gives up and tries the second one. The practical consequence is that when dns1 dies, a dig @10.8.0.8 proves dns2 is fine instantly, but ordinary apps on the laptop may see a few seconds of sluggish or failed lookups until mDNSResponder times out the dead server and settles on the healthy one, and until any stale negative entries expire from the various caches. This is not a flaw in the DNS layer. Both resolvers are genuinely up and answering. It is simply how macOS layers and manages its DNS caches. If it ever gets annoying during a real outage:

sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder

It clears the system cache and forces a fresh start.

Digression: on Linux (on the client-side) would be the same. Applications typically resolve names through getaddrinfo()/NSS rather than querying DNS servers directly. The actual resolver may be glibc using /etc/resolv.conf, systemd-resolved, dnsmasq, or another local stub. Multiple configured nameservers works as we all expected - the active resolver may keep using one server until a query times out or fails, then retry another. Local positive/negative caches and application-level DNS caches can add further delay. dig @<server> bypasses this resolver path and only verifies that the specified DNS server itself is reachable and answering.

So, back to the topic - the symmetric case (dns2 fails, dns1 serves) is identical. Either resolver can be lost, rebooted, upgraded or rebuilt without interrupting normal work. This also matters for routine maintenance and for operator error. I can push a change to one resolver, verify it, and only then touch the other, so a bad config never takes out both at once.

Both DNS servers fail

Name-based operation over the normal VPN becomes impaired. This is expected and, crucially, survivable. The infrastructure itself is still reachable by its known WireGuard IP addresses. Everything degrades gracefully:

DNS (both) are gone;
fall back to raw WireGuard IPs;
ssh bastion1 (will make a SSH connection to 10.8.0.3 from my laptop based on local ~/.ssh/config), then reach any host, repair DNS. If my local ssh-config evaporates - I have a traditional, Markdown docs with full topology (it's generated automatically, but it's a different topic, maybe for a different blogpost).

Main goal is - the DNS will be always a convenience layer. Losing it costs a little comfort, not access to anything.

The primary VPN hub (`vpn1`) fails, and so does `dns1`

Here the "listen on both hub addresses" detail earns its keep. The normal operational path is the vpn1 hub. If it goes down, the break-glass path is the second hub, vpn2, which reaches everything in the 10.9.0.0/24 subnet. Because each DNS host also listens on its vpn2-subnet address (dns2 on 10.9.0.8, for example), I can still reach and even query the surviving resolver over the emergency path:

# over the vpn2 break-glass path, dns1 is gone but dns2 answers
dig @10.9.0.8 monitoring1.int.example.com +short

10.8.0.5

But (AND this is the important distinction) I do not rely on that. Reaching DNS over vpn2 is a diagnostic convenience, not a dependency. The break-glass procedure is defined entirely in terms of raw IPs, so it does not care whether DNS is up or not.

Both DNS servers fail and `vpn1` fails, leaving only `vpn2`

This is the deepest survivable failure case. Everything comfortable is gone right now: no working internal DNS, no primary hub. What remains is the emergency path with the fewest possible dependencies:

turn on and connect to the vpn2;
ssh to the bastion by its 10.9.0.x IP;
ssh from there to any target by its 10.9.0.x IP;
diagnose and restore using raw addresses.

No name resolution is required anywhere in that sequence. The recovery plane was deliberately kept primitive. The single most important property of the whole system is that a catastrophic DNS failure can never, by itself, cut me off from the infrastructure.

Let’s push this further - a public upstream fails

Internal names under int.example.com keep resolving, because they are served locally and never leave the private network. Public names may be briefly unavailable until Unbound's RTT-based failover moves to another forwarder. Using three independent providers makes a total public-DNS outage extremely unlikely. If this happens, it means that humanity, including me, has much bigger problems... 🥲

Publicly trusted certificate for a private, internal services

Resolving monitoring1.int.example.com to 10.8.0.5 is only half the comfort. The other half is opening https://monitoring1.int.example.com:3000 and getting a green padlock with no warning, on macOS, iOS and every device/browser/script, without installing a private root CA on any device.

A quick word on why I use a sub-domain of a real domain at all, rather than a made-up suffix. There is in fact a proper, reserved name for exactly this purpose: home.arpa. Is defined by RFC 8375 as the standard domain for residential home networks (historically it grew out of the Homenet, RFC 7788, only ~10 years ago). It is the officially blessed way to name things on a home network without squatting on someone else's namespace or gambling on a future real TLD, and if I ever need internal resolution it is a perfectly good choice. Because I own my domain I can prove ownership of it to a public certificate authority, which home.arpa can never do (no CA will issue a publicly-trusted certificate for a reserved private name). That single property, ownership I can demonstrate over DNS, is what unlocks publicly-trusted TLS for private services.

The trick is that int.example.com sits under a real domain I own. That means I can get a normal, publicly-trusted Let's Encrypt certificate for monitoring1.int.example.com, even though that name only ever points at a private WireGuard address that is unreachable from the public internet. The certificate authority does not need to reach the service. It only needs proof, via DNS, that I control the domain. That is ACME DNS-01 validation. So I publish a TXT record under _acme-challenge.monitoring1.int.example.com. in the public example.com zone, and Let's Encrypt checks it.

So the architecture delivers, simultaneously:

Private network reachability.
PLUS private DNS records.
PLUS publicly trusted TLS certificates.

Everything above without ever exposing the service publicly, naturally.

Design choices for the SSL certificate

Two deliberate decisions I've had here.

First - per-host certificates or wildcard... So. Each monitoring host issues and holds only its own certificate for its own name. The private key is generated on that host by certbot and never leaves it, the same principle I already apply to WireGuard private keys. A wildcard *.int.example.com would mean one private key shared across all hosts fleet. Effect? Compromising any one host would then hand an attacker a key valid for every name in the namespace. Per-host certs keep the blast radius to a single machine.

Second decision - Grafana should serve HTTPS natively, with no reverse proxy (so with port :3000 at the end of the URL). Grafana can read a cert_file and cert_key directly and speak TLS on its own port. That keeps the stack minimal. The price is that the URL keeps its port, as I wrote, so https://monitoring1.int.example.com:3000, which I am happy to accept. No nginx, no Caddy, nothing new to run and secure.

The relevant part of grafana.ini, rendered only when TLS is enabled for that host:

[server]
protocol = https
domain = monitoring1.int.example.com
cert_file = /etc/grafana/tls/monitoring1.int.example.com.fullchain.pem
cert_key = /etc/grafana/tls/monitoring1.int.example.com.privkey.pem
http_addr = 10.8.0.5
http_port = 3000

Ansible owns everything around the certificate. So installs certbot, creates /etc/grafana/tls, installs a deploy-hook script that copies the issued fullchain.pem and privkey.pem into that directory with root:grafana ownership and mode 0640 and restarts Grafana, and it guards against enabling HTTPS before the certificate files actually exist (otherwise Grafana would refuse to start). What Ansible does not do is run the ACME challenge itself, because that step needs a human to publish a DNS record.

The configuration flow

On each monitoring host, reached through the bastion:

sudo certbot certonly --manual --preferred-challenges dns \
  -d monitoring1.int.example.com

certbot prints a TXT record to publish. I add it in the DNS provider's panel, wait for it to propagate, verify it against the domain's authoritative nameservers (not a public resolver, because that is what Let's Encrypt will check):

dig +short NS example.com
dig +short TXT _acme-challenge.monitoring1.int.example.com @<authoritative-ns>

and ONLY THEN press Enter. certbot issues the certificate and automatically runs any deploy-hook script sitting in its renewal-hooks/deploy/ directory, for any subcommand, not just certbot renew. So the deploy-hook I put there earlier fires on its own, right after this certonly call:

/etc/letsencrypt/renewal-hooks/deploy/monitoring1.int.example.com.sh

That copies the cert into /etc/grafana/tls, restarts Grafana, and the service comes up on HTTPS - no manual step needed.

If certbot instead prints "not due for renewal yet" and skips issuing a new certificate, the hook does not fire either, because nothing was renewed. In that case only, I run it once by hand:

sudo RENEWED_LINEAGE=/etc/letsencrypt/live/monitoring1.int.example.com \
  /etc/letsencrypt/renewal-hooks/deploy/monitoring1.int.example.com.sh

RENEWED_LINEAGE is the variable certbot itself would normally set when it runs the hook after a real issuance - running the script by hand means supplying it myself.

A quick check from the laptop, deliberately without the -k insecure flag, proves the served certificate is publicly trusted rather than a self-signed fallback:

curl -sI https://monitoring1.int.example.com:3000/api/health

HTTP/1.1 200 OK
(...)

If curl validates the chain against the system trust store with no flag, macOS and the browsers will too. Green padlock, no warning, no ugly "trust and visit anyway" exceptions...

The toil: manual renewal, and... What's next?

There is one genuinely annoying problem, and I will state it plainly. My domain registrar is one of the largest in the world, and some time ago (few years) they retired their public DNS REST API for all accounts. Without a working DNS API, I cannot automate the DNS-01 challenge. certbot's manual mode cannot renew unattended either: a --manual certificate issued without an authentication hook cannot be renewed by certbot renew, which is non-interactive by design. So renewal is not automated (yet). It is me, by hand, re-running the exact same certbot certonly command and re-publishing a TXT record, roughly every ~90 days. The directory-scanned deploy-hook then auto-fires on that re-issuance exactly as it did on first issuance (certbot 3.2.0+ version), copying the renewed cert into /etc/grafana/tls and restarting Grafana - no manual hook run needed unless certbot reported "not due for renewal yet" and skipped issuing.

To avoid the noise of a renewal timer that can never succeed, the setup masks the packaged certbot.timer, since it would otherwise try certbot renew twice a day and log failures forever.

This is a self-inflicted, temporary situation. The clean fix is either a registrar with a real DNS API, or delegating just the int.example.com sub-zone to a DNS provider that has one, so DNS-01 can be fully automated while the parent domain stays where it is 🎉. I will almost certainly migrate all my domains to a different operator in the next couple of months and be done with the manual ritual. It is, as problems go, a pleasant one to have - everything works today, and "the only cost" is a calendar reminder every couple of months.

Epilogue

Starting from two bare, hardened hosts, I now have a two-node, deliberately-equal Unbound DNS layer that serves internal int.example.com names from a single source of truth (the same mesh dictionary the rest of the fleet uses, expanded by an Ansible loop), caches and forwards everything else to three independent public providers, and binds only to private WireGuard interfaces behind three layers of access control. On top of that, Grafana on the monitoring hosts serves a publicly-trusted Let's Encrypt certificate over native HTTPS, reachable at a clean name from inside the VPN.

Now I can operate as follows:

1. Connect the VPN.
2. Now I am inside my infrastructure.
3. Names under int.example.com works, with valid TLS.
4. The normal internet works too.

And the safety net underneath it is smaller still:

something fundamental is broken;
then forget the convenience layers - use known WireGuard IP addresses;
reach the bastion and repair.

Convenience on top, a primitive and dependency-free recovery plane underneath. Keeping those two planes clearly separated is the whole point.