Blog

named was running on every node. It was serving nothing.

One certificate failed to issue on one server. The cause was on almost every other server too: BIND was up, the service check was green, and not one customer zone was being served. How a read-only audit found it, and how the fix avoided causing the outage it was fixing.

10 September 2026 · 8 min read

• DNS
• BIND
• Bash
• Linux
• Incident

Customers could host their DNS on our servers. Each server ran BIND, and each served the zones of the customers placed on it.

The first symptom was not a DNS alert. It was a certificate that would not issue on one server.

That is a DNS problem more often than it looks. Before Let’s Encrypt issues a certificate, it has to validate the domain — and every validation method starts with DNS. HTTP-01 has to resolve the name to find the server. DNS-01 has to read a TXT record out of the zone. If the authoritative server is not serving the zone, validation fails, and no certificate is issued.

My team lead fixed that server by hand. The question I was left with was the only one that mattered: is it just this one?

Running is not serving

The cause turned out to be in the template every server’s BIND configuration came from. /etc/named.conf was missing one line:

include "/etc/named/zones.conf";

zones.conf is the file that lists that server’s own customer zones. Without the include, BIND starts, loads its global options, listens on port 53 — and never reads a single customer zone. The logging block also expected a /var/log/named directory that did not exist.

Every health signal was green:

systemctl is-active named               # active
dig @127.0.0.1 example.com SOA +short   # (nothing)

systemctl answers is the process up? It cannot answer is it doing the job? The only real test of an authoritative DNS server is to ask it for a zone it is supposed to serve, and see whether it answers. An SOA answer means the zone is loaded. Nothing means it is not — no matter what the service manager says.

Measure the blast radius before changing anything

The first tool was deliberately read-only. For every active Linux server, the audit reported:

• is named running?
• does named.conf include zones.conf?
• does the log directory exist?
• which zones are configured on this server?

It changed nothing. The result: one server was correct — the original build node, whose configuration had quietly become the real reference — and nine customer domains were configured and not resolving.

The audit that stopped after one server

The first run of that audit reported on one server and then finished. No error.

The loop looked like this, in essence:

while read -r host; do
    ssh "$host" 'grep -c zones.conf /etc/named.conf'
done < servers.txt

ssh reads its standard input, so that it can forward it to the remote command. Inside a while read loop, its standard input is the loop’s input — the rest of the server list. The first ssh swallowed every remaining line, the loop’s next read found nothing left, and the loop ended normally.

The fix is one flag — ssh -n, which points ssh’s stdin at /dev/null — plus a comment in the script saying why, so nobody removes it. It is worth remembering beyond ssh: any command inside a while read loop that reads stdin will eat the list.

The fix could have caused the outage

Adding one include line to nine servers sounds like a two-minute job. It is not, because of one BIND rule: an include of a file that does not exist is fatal. named refuses to start.

Some servers had never had a zones.conf. On those, enabling the include would have turned a server that answered for nothing into a server that answered nothing at all — and would have done it with a restart, on a production DNS server, as part of a fix.

So the sync tool treats every step as something that can fail:

1. Skip a server that is already correct. The fix touches only what needs it.
2. Create an empty zones.conf if none exists — so that turning on the include cannot be fatal.
3. Back up the existing named.conf with a timestamp.
4. Install the reference config, owned root:named, mode 0640, and create the log directory owned by named.
5. named-checkconf — syntax and includes, before anything restarts.
6. Restart.
7. dig every configured zone, and treat a missing answer as a failure.

Any failure, at any step, restores the backup. The server ends in either the new state, verified, or its old configuration — never somewhere in between.

The line that must never be crossed

There is one rule in this work that matters more than the fix itself: zones.conf is per-server live data. It lists that server’s customer zones. Only /etc/named.conf is shared template content.

So zones.conf is never copied between servers, and never managed by a template. Copy it from the reference node to a sibling, and you replace that server’s customer list with a different server’s. Put it under configuration management, and the next run overwrites every server’s list with the template’s — wiping live domains while reporting success.

That distinction — which files are template, and which are data — is the thing to get right before automating anything that touches them.

Result

Two findings were outside the task, and were reported rather than “fixed” on the way past. Three zones on one server still did not answer — stale entries for domains since mapped elsewhere, which my team lead confirmed and kept for himself. And one zone logged a mail exchanger pointing at a CNAME, which the DNS standards forbid and which will affect mail routing — a zone-data problem for its owner.

Fixing the fleet also does not stop a new server repeating the fault, because new servers are built by a provisioning playbook from the same stale config. That change went to the playbook’s owner as a finished task — with an explicit warning that the playbook must never manage zones.conf or the zone directory.

I also tested the zone tooling’s own guards while I was there: a CNAME at a zone apex was refused before any zone was generated, and a zone that failed named-checkzone was rejected and set aside while the live configuration stayed clean.

What I took from it

• A process being up proves nothing about the service. For DNS, the test is a query. For anything, the test is the job itself.
• One failure is a sample, not a diagnosis. The fix on one server was correct; the question it raised was about the other nine.
• Audit read-only first. Knowing the blast radius before changing anything turned “fix DNS” into a list of seven specific servers.
• Make the fix unable to cause the outage. Prepare the preconditions, validate, keep a backup, roll back on any failure.
• Separate template from data before you automate. Automation that cannot tell them apart will eventually overwrite the data with the template.

---

This came out of the SSL certificate work — a certificate that would not issue on one server was the first visible symptom of a DNS fault on almost all of them.

References

• BIND 9: the include statement
• BIND 9 manual pages — named-checkconf validates configuration syntax and includes; named-checkzone validates a zone file.
• RFC 8555 §8: ACME identifier validation — HTTP-01 and DNS-01 both depend on DNS.
• RFC 1034 §3.6.2 — a CNAME cannot coexist with other data, so it can never sit at a zone apex.
• RFC 2181 §10.3 — an MX must not point at an alias.
• ssh(1): -n — redirect stdin from /dev/null; required when ssh runs in the background or in a loop.

← All posts