SSL Certificate Lifecycle Automation

Production system at a previous employer. The screenshots are real, with customer domains and the control node's hostname redacted. The founding principle, that one isolated process owns all key material, was my team lead's design. The scripts described here are mine.

The problem

A fleet of shared hosting servers, each serving many customer domains through nginx, with certificates from Let’s Encrypt. A central repository on the control node keeps a copy of every certificate so a domain can move between servers without being reissued.

That last capability is where it broke. Certificates moved to a new server served perfectly — valid padlock, no errors, nothing in any log — and then silently failed to renew about ninety days later. The same failure turned up across 26 domains.

There was a second, quieter problem underneath: certificate handling had grown up piece by piece, so key material was touched by more processes than it needed to be. My team lead’s direction was one isolated, root-owned process that handles all of it, so that the web tier never touches a private key. I wrote that process, and the node-side scripts it hands work to.

Why the certificates stopped renewing

A certbot certificate is not four files. It is a lineage:

The old transfer copied the four PEM files straight into live/ as real files. nginx does not care whether it reads a symlink or a file, so the site served normally. certbot does care. It no longer recognised the directory as a lineage it owned, so the renewal run skipped it — without an error, because from certbot’s point of view there was nothing to renew.

A certificate that works today and fails in ninety days is the worst kind of defect to have, because every check you run on day one passes. That is the same shape as the rest of this site’s write-ups: configured, visible, green, and inert.

It had a second edge. certbot appends -0001, -0002 when it cannot reuse a lineage name, so one domain can own several lineages. Anything that looks only at the plain domain name misses the rest — confirmed on a live domain carrying two.

What I built

cert_repo.py — the repository owner

Nothing else reads, writes or transfers certificate material. Callers ask this script, and it decides. Four verbs:

The decisions that matter:

• Private keys go on stdin, never argv. A command line is readable by every user on the host through ps and /proc.
• Exit 1 is an answer, not an error. It means “no usable certificate here — issue a new one”, and the caller branches on it. Exits 2–5 are the real failures, each distinct: configuration, refused input, transfer failed, repository unwritable.
• One JSON object on stdout, always. Progress goes to stderr. The domain is merged into every response, so no failure path — however early — comes back without saying which domain it was about.
• A certificate is released only to the server it is assigned to. The assignment was recorded at issuance; what was missing was the check that enforced it at transfer time.
• Transfer never writes into certbot’s tree. Files land in a fresh, timestamped staging directory on the node, so the node never sees a partial set; the key is chmod 0600 on arrival; three retries, then a real TRANSFER_FAILED rather than a hopeful one.

What the transfer replaced

One line of the old path, which is worth reading slowly:

• the destination path was interpolated into a shell command
• the server’s ID was used as its hostname
• the copy’s exit status was ignored, so it reported success unconditionally
• and it wrote real files into live/ — the lineage bug above

Four defects in one line, and the third is the one that hid the other three: a transfer that always reports success cannot tell you that anything else is wrong.

install_cert.py — building a lineage, not copying files

Takes what cert_repo.py staged and installs it the way certbot would have:

1. Validate the staged files — nothing is touched until they pass
2. Write them into archive/ as the next version number
3. Record where the live/ symlinks currently point, then repoint them
4. Preserve the existing renewal config — it carries this node’s ACME account, and overwriting it would orphan the account
5. nginx -t, then reload — and if nginx refuses, put the previous symlinks back and discard the new archive version
6. Ask certbot itself whether it recognises the lineage, and report the answer

That last step deliberately does not check the files. The installer’s own opinion of the lineage is the thing under test, so it asks the tool whose opinion actually decides renewal.

A live/ directory already full of real files — the broken state — is not deleted. It holds the certificate the node is serving at that moment, so it is moved to a timestamped quarantine, where it can be restored.

lineage.py — one definition, shared

Every script that touches certificates imports the same module for what a lineage is: how to enumerate numbered lineages, how to find a healthy one, how to detect half-deleted leftovers, how to quarantine and restore. Four scripts with four private ideas of what a lineage is would have drifted apart; one module cannot.

nginx_ssl.py — disabling HTTPS without deleting anything

nginx_ssl.py <domain> disable|enable [--dry-run]

The entire mechanism is moving files. nginx loads conf.d/*.conf, and that glob does not recurse — so a config moved into conf.d/keep/ simply stops being read. Each domain’s HTTP config lives in a separate file and is never touched, so the site keeps serving on port 80 throughout.

• All four filename forms an SSL config can take are handled together, including per-subdomain variants
• nginx -t before reloading; if nginx refuses, every file is moved back before the script reports
• Renewal is left running, so a disabled domain keeps a current certificate and re-enabling is a move, not a reissue
• enable restores the newest timestamped set and refuses to overwrite a file that already exists

revoke.py — fixed on my team lead’s instruction

The certificate was always revoked at the CA correctly. The local cleanup after it was not:

• it removed one SSL config file and left every subdomain config in place, pointing at a revoked certificate — now all four forms move to keep/
• it deleted the lineage’s archive outright, destroying every historical key — now the whole lineage is quarantined with a timestamp
• it had no validation on the domain argument, which went straight into path construction — an empty value would have collapsed the target path onto its parent directory

The principle behind all three is my team lead’s, and it applies estate-wide: never delete — move, with a datetime signature.

The bridge to the firewall — manage_https_firewall.sh

Certificates and the firewall meet at port 443, and this script is where. Issuing a certificate opens 443 on that server; losing one may close it.

May, because iptables works at the TCP layer and the hostname lives in the application layer. Port 443 is open or closed per server, never per domain. Closing it because one domain lost its certificate would break HTTPS for every other site on that host. So remove counts the server’s remaining active certificates first and declines if any remain — and that guard sits on the action, so revocation, deactivation and expiry all pass through it.

If the count cannot be established at all, the script refuses to close the port and exits 3. That asymmetry is deliberate: a port left open with no certificate behind it serves nothing, whereas a port closed under live certificates is an outage.

add is state-based rather than timed — it checks whether 443 is already open and skips the deploy if so. Provisioning issues many certificates to one server in quick succession, and only the first should pay for a firewall deploy.

Its server-ID validation caught a real typo during live testing: an ID with an extra digit, which would otherwise have written a rule scoped to a server that does not exist — correct-looking in the table, matching nothing, and leaving 443 silently closed.

This connects to the firewall platform, which is where that rule is compiled and deployed.

Every passing test was talking to a stub

Testing the full chain on a node — rather than trusting what the control plane reported — turned this up.

The node endpoint responsible for installing a transferred certificate returned Certificate synced. — and did nothing else. The real implementation existed in the deployment template; that node had simply never been redeployed with it.

So every earlier “end-to-end test passed” had been a conversation with a stub. The control plane recorded success. No certificate was installed.

It is the same failure as the backup check that verified nothing: a verification step that is green because it is not doing anything. The response I took from it is the same too — assert on the effect, never on what the API says. After an install, live/ must contain symlinks and certbot certificates must list the lineage. A success message is not evidence.

Hardening the deployment itself

The scripts run as root, so how they are deployed is part of the security design:

• Template scripts were world-writable — one at 0777, one group-writable by the web server’s user — while being deployed fleet-wide and executed as root. Fixed: anyone who could write those files effectively held root on every server that ran them.
• The installer’s blanket chmod -R 755 caught the staging directory, and the private keys inside it. Fixed — staging is pruned from the recursive chmod and created at 0700.
• sudo rules name individual people, not a group. The obvious group had thirteen members including the web server’s user. The caller list is enumerated so that adding someone is a deliberate act.
• Each sudo command form is pinned separately. sudo concatenates arguments into one string before matching, so a trailing * already covers any argument list — and a literal placed after a wildcard makes position matter. The rule allows exactly the forms that are called, and no others.
• The installer proves its own result. It validates the rendered sudoers file before installing it, and finishes by checking that the web server’s user can actually run the scripts through the exact interpreter the rule names.

Retention

Certificate material the business no longer needs is liability, not backup. Thirty-day retention — my team lead’s figure — was applied: 16 domains and 9 archived entries retired.

It runs weekly, not daily, on purpose. The material is months old by definition, and a slower cadence leaves a wider window to notice a mistake before the next run compounds it. It only removes domains whose status is inactive; a directory it cannot match to a known domain is reported, never deleted; and every removal is logged before it happens.

Design decisions

A lineage is the unit, not a file. Every script reasons about certificates the way certbot does, through one shared module, because certbot’s opinion is what decides whether a certificate renews.

Reversible by default. Configs move to keep/, lineages move to a timestamped quarantine, prune is a dry run unless told otherwise. Every destructive operation in this system can be undone by moving something back.

Refuse rather than guess. Unknown state → exit 3 and leave the firewall alone. Unmatched directory → report it. nginx rejects the result → roll back before reporting. The safe failure is the one that changes nothing.

Machine-readable, exactly once. One line of JSON on stdout, detail in the log — so a caller that merges stderr into stdout still gets one parseable object.

What was left when I handed over

Fleet rollout. The full chain — transfer, install, nginx reload — was verified end to end on one reference node. Rolling it out to the rest of the fleet was the first item in my handover document. Everything I have described is proven on one machine; I would rather say that than imply otherwise.

Host key pinning. Transfers trust an unknown host key on first connection and refuse a changed key on a known host — and each transfer reports which case applied. I named that trade-off to my team lead at the time. At scale I would pin host keys when a node is enrolled rather than trust on first use.

Multi-lineage cleanup — domains holding several numbered lineages from earlier tests — was parked by my team lead as a separate piece of work.

What I would do differently

Build the end-to-end assertion first. The stub would have been found on day one by a test that checked live/ for symlinks instead of reading the API’s reply. Finding it took deliberate testing; an automated assertion would have found it by default.

Make the renewal failure observable. Ninety days is a long time for a defect to stay invisible. A daily check that every served certificate belongs to a lineage certbot recognises — and alerts when one does not — would have turned 26 silent failures into one alert on the first day.