Blog

Five ways to verify a fix without testing it

A passing check, a clean query, a plausible error, a success message. Each of these looked like evidence, and each was measuring something other than the change I thought I was testing. The habits I use now so that "verified" means something.

· 8 min read

  • Debugging
  • Python
  • MySQL
  • Linux
  • Testing

Most of the posts on this site are about a control that looked healthy and did nothing: a backup check that checked nothing, a firewall service that was enabled and never ran, a DNS server that was up and served no zones.

This one is about the same failure in my own process. Every item below is a way a check looked like proof that a change worked, when it was not — and the ones that happened cost real time. Each was real evidence, of something else.

WHAT IT LOOKED LIKE WHAT IT ACTUALLY PROVED 1 · I changed the orchestrator and examined the results a copy nothing executes was edited cron ran a file somewhere else 2 · the results table looks right after the restart rows exist — not which code made them any that finished before the restart are old 3 · rowcount = 0 so the row isn't there no row was changed it may have matched and already held the value 4 · 407 from the proxy vendor so they are rejecting our credentials the credentials were "<USERNAME>" a placeholder, URL-encoded and sent 5 · "Certificate synced." so the certificate is installed an endpoint returned a string a stub; nothing was installed
Every left-hand box is true. None of them is the claim it was used to support.

1. I edited a file that nothing runs

There were two similarly named copies of the proxy checker’s orchestrator, in two directories. I worked in one of them for several sessions — changes made, results examined — before a grep of /etc/cron.d showed that the job actually running was pointed at the other.

Every conclusion from those sessions was about code that was not executing. Nothing errored, because the file I was editing was perfectly valid. It just was not the one anybody ran.

The habit now: on an unfamiliar system, before opening an editor, ask the system what it is actually running.

systemctl cat <unit>                    # the real ExecStart, with drop-ins
grep -rn '<script-name>' /etc/cron.d /etc/crontab /var/spool/cron 2>/dev/null
ps -eo pid,args | grep '[o]rchestrator' # what is running right now

2. The results can be older than the fix

Deploy a change, restart the service, query the results table — and the rows look right. The trap is that nothing in a result set says which code produced it. Rows that finished before the restart came from the old code, and they look exactly like rows from the new one.

A query returns what is in the table, not what was produced since you last touched something. Recent-looking rows are not necessarily rows from this deployment.

The habit now: select the timestamp, and compare it with the moment the new code started.

systemctl show -p ActiveEnterTimestamp <unit>
SELECT id, status, finished_at FROM queue
WHERE finished_at > '<ActiveEnterTimestamp>' ORDER BY finished_at DESC;

If no row is newer than the restart, there is no evidence yet — only a table.

3. rowcount said nothing happened

It is tempting to use an UPDATE as a probe: set a column to the value it should already have, and read rowcount to see whether the row is there. When rowcount comes back 0, it looks like the row is missing.

It does not mean that. MySQL’s affected-rows count for an UPDATE is, by default, the number of rows actually changed — not the number the WHERE clause matched. A row that already holds the value is matched and not changed. SET x = x reports 0 for every row it touches, and so proves nothing about any of them.

The habit now: to know whether a row exists, SELECT it. If you genuinely need matched rows from an UPDATE, ask for them — the CLIENT_FOUND_ROWS connection flag (client_flag=CLIENT.FOUND_ROWS in PyMySQL) changes the count to rows found. Know which one your driver is giving you before you read anything into it.

4. The vendor rejected a placeholder

A proxy check failed with 407 Proxy Authentication Required. That is exactly what a vendor sends when it rejects your credentials, so that is what it looked like — and it cost two debugging sessions to establish that it was not.

The test payload had been written with placeholders for the credentials, meaning to fill them in:

{ "proxy_username": "<USERNAME>", "proxy_password": "<PASSWORD>" }

They were never filled in. The checker built a proxy URL from them — URL-encoded, < and > become %3C and %3E — and sent it. The vendor correctly rejected a user called <USERNAME>. A perfectly plausible error, for a completely different reason.

The same thing happens with commands. <IP> written into a command for someone else to run does not fail as “you forgot to substitute this”; it fails as Expecting value: line 1 column 57, which sends everyone looking at JSON.

The habit now: never put a placeholder inside something executable. Test credentials are read from the database by the test itself, so nothing is typed — which also keeps real secrets out of shell history, chat and documents. And commands written for others are correct exactly as written, because the person running them will, correctly, paste them literally.

5. The API said it worked

The last one was not my code, but it fooled everyone who tested against it, and it is the most instructive. On the certificate platform, the endpoint on each server responsible for installing a transferred certificate returned Certificate synced. — and did nothing else. The real implementation existed in the deployment template; that server had simply never been redeployed with it.

So every earlier “end-to-end test passed” had been a conversation with a stub. The control plane recorded success. No certificate was installed.

It was found by testing the chain on the server itself, rather than trusting what the control plane reported.

The habit now: assert on the effect, never on the response. After an install, the certificate’s live/ directory must contain symlinks, and certbot certificates must list the lineage. A success message is a claim, not evidence.

The pattern

What was checkedWhat it provedWhat would have proved the claim
the file I editedthat file changedthe file cron or systemd executes
rows in the tablerows existrows finished after the restart
rowcounthow many rows changeda SELECT of the row
the error messagethe vendor rejected somethingthe payload actually sent
the API responsethe endpoint returned a stringthe certificate on disk, and certbot’s view of it

In every case the check was one step removed from the thing being claimed: a copy instead of the running file, a table instead of this run’s output, a count of changes instead of a count of matches, an error instead of the request, a response instead of the state.

The fix is the same each time. Before calling something verified, ask: what exactly would this check look like if my change had done nothing? If the answer is “the same”, it is not a verification.

Items 1–4 came out of the distributed proxy checker; item 5 out of the SSL certificate automation.

References

← All posts