Blog

Why copied certificates stop renewing ninety days later

Copy a Let's Encrypt certificate's files to another server and it serves perfectly — padlock, no errors — right up until it expires, because certbot no longer treats it as a certificate it owns. We found that across 26 domains. Here is what a certbot lineage actually is.

4 September 2026 · 9 min read

• TLS
• Let's Encrypt
• certbot
• nginx
• Python

A fleet of shared hosting servers, each serving many customer domains, with certificates from Let’s Encrypt. A central repository keeps a copy of every certificate, so that when a domain moves to a different server it can take its certificate with it instead of being reissued.

The transfer did what you would expect. It copied the certificate, the chain and the private key into /etc/letsencrypt/live/<domain>/ on the new server, and nginx picked them up. Valid padlock. No errors. Nothing in any log.

About ninety days later, those certificates expired. The same failure turned up across 26 domains.

Nothing was wrong with the files. What was wrong was the assumption that a certificate is its files.

A certificate is a lineage

certbot does not store a certificate as four files in a directory. It stores a lineage — three pieces, in three places, that only work together:

A renewal writes cert3.pem into archive/ and repoints the symlinks in live/. Your web server config points at live/, so it picks up the new certificate on the next reload without anyone editing anything. certbot’s own documentation says it plainly: “Rather than copying, please point your (web) server configuration directly to those files.”

nginx does not care about any of this. It opens a path and reads bytes. A symlink and a regular file are the same to it. certbot cares a great deal.

How copying breaks it — two ways

What happens next depends on what arrived with the files.

No renewal config on the new server. certbot has no record that the certificate exists. certbot renew runs on its schedule, finds nothing to do for it, and exits cleanly. There is no error, because from certbot’s point of view there is nothing to renew.

A renewal config, but real files in live/. certbot’s storage code checks that each of the four live/ paths is a symlink into archive/, and raises expected … to be a symlink when one is not. The renewal run logs “Renewal configuration file … is broken”, skips it, and exits non-zero at the end.

That second case is not silent — certbot says exactly what is wrong. But it says it in the output of a timer or cron job, on one server of many, in a log nobody reads until a customer reports an expired certificate. Loud in a place nobody is listening is the same as silent.

The same domain, more than once

There was a second edge. When certbot cannot reuse a lineage name, it appends a counter: example.com-0001, example.com-0002. So one domain can own several lineages at once — and anything that looks only for live/example.com/ misses the rest. That was not hypothetical: a live domain on the reference node had exactly two.

And there is a third state, which appears when cleanup is done by hand: a half-deleted lineage. Remove only the renewal config, and the next certbot certonly --cert-name <name> decides it is creating a brand-new lineage, finds the archive directory already there, and aborts with archive directory exists for <name>.

So the shared module every certificate script imports, lineage.py, knows how to enumerate numbered lineages, find the newest healthy one, detect half-deleted leftovers — and move them aside rather than delete them, because archive/ holds private keys.

Building a lineage instead of copying files

The fix was an installer, install_cert.py, that takes what the repository transferred and builds the lineage certbot would have built itself:

1. Validate first. Nothing on the node changes until the staged files pass: all four parts present, the certificate readable, covering the domain and not expired — and the private key’s public half identical to the certificate’s, which is the cheap, definitive test that the two belong together.
2. Write into archive/ as the next version number, with certbot’s own permissions: the archive private, the key more so.
3. Record where the live/ symlinks point now, then repoint them — as relative links, exactly as certbot writes them, so the tree can be moved.
4. Keep an existing renewal config. It carries this node’s ACME account; overwriting it with the transferred one would orphan the account. Only when there is none is the transferred config used, with its paths rewritten for this node.
5. nginx -t, then reload. If nginx refuses, the previous symlinks go back and the new archive version is discarded.
6. Ask certbot. certbot certificates --cert-name <domain> — does certbot recognise what was just built?

That last step is deliberate. The installer’s own opinion of the lineage is exactly what is under test, so it asks the tool whose opinion actually decides renewal.

A live/ directory already full of real files — the broken state — is not deleted either. It holds the certificate the node is serving at that moment, so it is moved to a timestamped quarantine, where it can be restored.

One footgun on the way

Step 4 rewrites paths in the renewal config with a regular expression. The first version built the replacement as a string — in essence:

re.sub(r"(archive_dir\s*=\s*).*", rf"\g<1>{new_path}", text)   # wrong

re.sub parses escape sequences in a string replacement, so a backslash in the inserted path is interpreted rather than inserted. A replacement function receives its return value verbatim:

text = re.sub(r"(archive_dir\s*=\s*).*",
              lambda m: m.group(1) + str(ARCHIVE_ROOT / domain),
              text)

It is the kind of bug that only appears when the input contains the character you never tested with — which, for a path, is only a matter of time.

What I would add: make the failure observable on day one

Ninety days is a very long time for a defect to stay invisible. A daily check that every certificate nginx serves is a symlink into a lineage — and alerts when one is not — would have turned 26 expiries into one alert on the first day:

# every certificate nginx references must be a lineage symlink, not a copied file
grep -rhoE 'ssl_certificate\s+[^;]+' /etc/nginx/conf.d | awk '{print $2}' | sort -u |
while read -r f; do
    [ -L "$f" ] || echo "NOT A LINEAGE: $f"
done

\s+ straight after ssl_certificate is what keeps ssl_certificate_key lines out of the list.

What I took from it

• “It serves” is not “it is managed”. The web server and the renewal tool have different ideas of what a certificate is, and only one of them decides whether it still works in three months.
• Build state the way the owning tool builds it — or ask the tool to do it. Hand-assembled state that looks right to one consumer can be invisible to another.
• Verify with the tool that decides. Not “the files are there”, but “certbot lists it”.
• Never delete key material to fix a mess. Quarantine it with a timestamp. The cleanup that deletes the wrong thing is the one you cannot undo.

---

This came out of the SSL certificate lifecycle automation, where the installer, the repository owner and the shared lineage module are described in full.

References

• certbot: where are my certificates? — live/ holds symlinks; point your server at them rather than copying.
• certbot storage.py — _check_symlinks(): “expected … to be a symlink”.
• certbot renewal.py — a lineage that fails to load is logged as broken and skipped.
• RFC 8555: ACME — the protocol behind the account a renewal config carries.
• Python re.sub — escapes in a string replacement are processed; a function’s return value is not.
• nginx ssl_certificate

← All posts