Blog

The rollback watcher that held the lock for five minutes

A dead man's switch is the right way to change the firewall on a machine you cannot physically reach. Mine worked — and for five minutes after every run, it also made every new refresh fail with "already running". A file descriptor went somewhere I never sent it.

· 8 min read

  • Bash
  • Linux
  • iptables
  • Reliability

Applying a firewall to a remote machine is one of the few operations that can permanently disconnect you from the machine you are changing. Get one rule wrong and your SSH session is the last one anybody gets.

The standard defence is a dead man’s switch: before applying a change, arm something that will undo it unless you confirm in time. If the change locks you out, you cannot confirm, and the machine puts itself back.

This is how the firewall refresh on the control nodes worked, what made it safe — and the bug that made it refuse to run for five minutes after every success.

The switch

In order:

  1. Back up the live ruleset, plus the structured file it was loaded from.
  2. Validate the new ruleset with iptables-restore --test.
  3. Arm the watcher — a background process that sleeps for the rollback window, then restores the backup unless a confirmation marker exists. It is armed before the apply, so even an apply that hangs halfway is covered.
  4. Apply the new ruleset.
  5. Verify that access survived: the administrators’ rules are live in the kernel, and sshd is still listening.
  6. Confirm — only if verification passed.

The window was 300 seconds on the main control node and 180 on the other. Managed nodes had a separate watchdog, run from cron every minute, with a 120-second grace period.

Three details made it trustworthy rather than merely present:

Confirmation is earned, not assumed. The script confirms because access was verified, not because iptables-restore returned 0. A ruleset that loads cleanly and locks everyone out also returns 0.

Your own session proves nothing. Connection tracking lets an established SSH session survive almost any ruleset. “I’m still connected” says nothing about whether a new connection would work — which is why manual confirmation is only valid from a genuinely new session.

Confirmation markers are scoped to the run. With one shared marker file, two applies inside the same window interfere: run B confirms, run A’s watcher sees B’s marker and keeps A’s change, and then B’s own watcher finds the marker already consumed and rolls B back. Each run now has its own marker, named after its run ID, so no run can satisfy or starve another’s watcher.

The lock

Two refreshes at once would race each other’s watchers, and the UI can trigger a refresh from several sessions. So the script serialises itself, and refuses rather than queues:

exec 9>"$LOCK"
if ! flock -n 9; then
    err "another refresh is already running - refusing to run concurrently"
    exit 75
fi

File descriptor 9 is opened on the lock file, and flock takes an exclusive lock through it. When the script exits, fd 9 closes, and the lock is released.

That last sentence was the bug.

“Another refresh is already running”

After a refresh succeeded, every refresh started in the next five minutes failed with that message. Nothing was running. The previous refresh had finished, confirmed its change, and exited cleanly.

Here is what was still alive: the watcher. And the watcher had fd 9.

0 s ~10 s 60 s 300 s refresh_master.sh fd 9 · lock taken exits apply_master_rules.sh inherits fd 9 ( sleep 300 … ) & rollback watcher · inherited fd 9 · lock still held next refresh flock -n 9 fails → exit 75 Nothing is running a refresh. The lock is held by a process that only sleeps. WITH 9>&- ON THE APPLY COMMAND refresh_master.sh lock taken → released apply + watcher no fd 9 — the watcher still guards the change, but holds no lock
The lock outlived the process that took it because a child it never knew about kept a copy of the descriptor.

A flock lock does not belong to a process. It belongs to an open file description, and every file descriptor that refers to that description refers to the same lock. The man page is exact about when it goes away: “the lock is released either by an explicit LOCK_UN operation on any of these duplicate file descriptors, or when all such file descriptors have been closed.”

All of them. And a child process inherits every open descriptor its parent has. So:

  • the refresh script held fd 9,
  • the apply script it started inherited fd 9,
  • the watcher that the apply script put in the background — a subshell that sleeps for the whole window — inherited fd 9 too.

The refresh script exited. The apply script exited. The watcher slept on, holding a descriptor it never used, and with it the lock, for the full 300 seconds. Every refresh in that window found the lock taken and correctly reported that another one was running.

Every component behaved exactly as documented. The lock worked. The watcher worked. They were simply joined by something neither of them was looking at.

The fastest way to see a problem like this is to ask who has the file open — lsof or fuser on the lock file — or to look at /proc/<pid>/fd for the sleeping process. A lock whose holder you have not identified is a guess, not a diagnosis.

The fix is four characters

# 9>&- closes our lock fd for the child. apply_master_rules.sh backgrounds a
# rollback watcher that sleeps for the whole window, and a subshell inherits
# every open descriptor - including fd 9.
FW_RUN_ID="$run_id" ROLLBACK_DELAY="$ROLLBACK_DELAY" \
    /usr/bin/bash "$APPLY" "$RULES" 9>&- >>"$LOG" 2>&1

9>&- is a redirection that closes descriptor 9 for that one command. The apply script — and therefore its watcher — starts without the lock descriptor. The lock stays with the refresh script, which is the only process that should hold it, and it is released when that script exits. The watcher still guards the change for the full window; it just no longer holds a lock while it does.

Two alternatives do the same job:

  • flock -o, when flock runs the command itself. The man page describes that option as being for exactly this case: “useful if command spawns a child process which should not be holding the lock.”
  • Closing the descriptor in the background job, ( exec 9>&-; sleep … ) &. It works, but it puts the fix in the child, which has to know about a lock it did not take. The parent is the one that knows what it opened.

What I took from it

  • A lock belongs to a file description, not to the process that took it. Anything that inherits the descriptor holds the lock.
  • Background jobs inherit everything — descriptors, environment, working directory. A long-lived background child is the easiest place for something to outlive its purpose.
  • Close what a child should not have, at the point you start it. The parent knows what it opened.
  • Test the second run, not just the first. The first refresh always worked. The bug only existed for whoever came next.

This is part of the safe-apply path in the firewall automation platform. The verification step in the same flow had its own bug.

References

  • flock(2) — locks are tied to open file descriptions, preserved across fork and execve, and released when every duplicate descriptor is closed.
  • flock(1) — -n, and -o for commands that spawn children which should not hold the lock.
  • Bash manual: duplicating file descriptors — n>&- closes descriptor n.
  • open(2) — file descriptors versus open file descriptions.
  • iptables-restore(8) — --test, used before anything is applied.

← All posts