Blog

"All tables OK" — the backup check that verified nothing

My restore verification printed a green result immediately after the binary it depends on failed to exist. Here is how a shell fallback turned a hard failure into a success message, and what it cost to find.

27 August 2026 · 8 min read

• MySQL
• Bash
• Backups
• Disaster Recovery
• Docker

I had just finished the part of a project I was most pleased with: a script that pulls the latest database backup from S3, restores it into a throwaway container, and checks that every table survived. A backup you have never restored is a hypothesis. This was supposed to be the thing that turned it into a fact.

It printed this:

[2026-08-27T12:42:10Z] Running integrity check across all restored tables
[2026-08-27T12:42:10Z] All tables OK
[2026-08-27T12:42:10Z] RESTORE VERIFIED — recovery time 1s

Directly above those lines, in the same terminal, was this:

OCI runtime exec failed: exec: "mysqlcheck": executable file not found in $PATH

The integrity check had not run. The binary did not exist. And the script reported success.

The line that did it

docker exec "$CONTAINER" mysqlcheck -uroot -pverify --all-databases --check \
  | grep -v 'OK$' || log "All tables OK"

Read innocently, that says: run the check, filter out the lines ending in OK, and if nothing is left over, say everything was fine. Which is what it does — when mysqlcheck exists.

mysqlcheck is not in recent mysql:8.0 images. So the sequence was:

1. docker exec fails and writes its error to stderr, which is not part of the pipe
2. Nothing goes into grep at all
3. grep on empty input finds no matches, so it exits 1
4. || sees a non-zero exit and fires the fallback
5. The fallback prints All tables OK

Every step behaved exactly as documented. The bug is that grep exiting 1 was being read as “no problems found” when it actually meant “no input arrived.” Those two states are indistinguishable to grep, and I had written code that could not tell them apart.

WHAT I ASSUMED mysqlcheck rows of output grep -v ‘OK$’ verified

WHAT HAPPENED docker exec binary not found

empty pipe error went to stderr grep exits 1 "no matches" || fallback "All tables OK"

Both paths end in a green line. Only one of them checked anything. grep cannot distinguish “nothing was wrong” from “nothing arrived”.

Why this one mattered more than the others

I found six defects in that project by running the code rather than reading it. Five were ordinary: a handler that could not be registered, a task that only worked on first run, a WAF rule that matched nothing. Annoying, findable, fixed.

This one was different, because it is the exact failure the project existed to prevent.

The whole argument for the restore script is that an untested backup is a guess. But a test that silently no-ops is not better than no test — it is worse, because it produces evidence. It writes a green line into a log. It fills in a row in a results table. It lets you tell a client that restores are verified nightly, in good faith, while nothing is being verified at all.

A backup you have never restored is a hypothesis. A restore whose integrity check silently no-ops is the same hypothesis wearing a tick mark.

The fix, and the second bug in the fix

I replaced mysqlcheck with CHECK TABLE, driven from information_schema and run through the mysql client, which is always present in the image. The new version fails loudly on empty output and prints how many tables it checked.

Then it failed the opposite way.

Because the replacement captures stderr deliberately — so that a real error cannot vanish the way the first one did — it also captured this:

mysql: [Warning] Using a password on the command line interface can be insecure.

That line does not end in OK. So it was reported as an integrity failure. The check had actually passed.

Two attempts, two opposite failures:

Attempt	Failure	What it showed
mysqlcheck + || fallback	False negative	Green while checking nothing
CHECK TABLE + stderr capture	False positive	Red while everything was fine

Same root cause both times: pattern-matching a mixed stream without controlling what is in it. In the first case stderr escaped the pipe entirely. In the second it joined the pipe and polluted it. Neither version knew what it was actually reading.

The real fix was not a better pattern. It was passing the password through MYSQL_PWD so the warning is never emitted at all — removing the noise at source rather than filtering it downstream.

What I would take from this into any script

Distinguish “no findings” from “no input.” Any pipeline that ends in grep, awk or wc and treats emptiness as success has this bug. Count what you processed and assert on the count:

CHECKED=$(printf '%s\n' "$OUT" | grep -c .)
[[ "$CHECKED" -eq 0 ]] && { log "FATAL: integrity check produced no output"; exit 1; }
log "All tables OK (${CHECKED} checked)"

Be careful what || is catching. cmd | filter || fallback runs the fallback on the filter’s exit code, not the command’s. If you want the command’s status, set -o pipefail changes that — and had I set it here, this bug would have surfaced on the first run.

Verify the verifier. The cheapest test I never wrote: delete a table from the dump and confirm the check goes red. Any assertion you have never seen fail is not yet an assertion — it is a hopeful print statement. Same reason I now write an alert that fires when monitoring stops scraping, not just when the service breaks.

Prefer tools that are actually installed. mysqlcheck was absent because upstream images have been trimming client utilities for years. The mysql client and CHECK TABLE are not going anywhere.

The wider version of this

This is a specific bug in a specific script, but the shape of it is everywhere:

• A WAF rule deployed and green in the dashboard, matching nothing, because the field it inspects is the raw query string and the pattern assumed it was decoded
• A ServiceMonitor that Prometheus ignores because a selector default excluded it, so a target simply never appears
• A NetworkPolicy accepted by the API server on a cluster whose CNI does not enforce them

In every case the control is configured, visible, and inert. Nothing errors. The dashboard is green.

Controls that fail loudly are worth more than controls that work quietly, because the second kind eventually stops working and does not tell you. The only defence I know of is to make each control prove itself — trip it on purpose, watch it go red, and write down what you saw.

---

This came out of building a multi-AZ web platform with MySQL replication and verified backups, then deliberately breaking it. The full write-up has the measured recovery numbers and the other five defects.

References

• grep exit status — 0 for a match, 1 for none, 2 for an error. The distinction that matters here.
• Bash pipefail — makes a pipeline return the failing command’s status rather than the last one’s.
• MySQL CHECK TABLE
• MySQL environment variables — MYSQL_PWD and why the client warns about -p.

← All posts