Blog

grep -q found the match. pipefail called it a failure.

A check that passed all morning decided at 10:14 that nothing was listening on port 22, and rolled back a healthy firewall. It was the second time the same two words had bitten me. Here is the race behind both.

· 7 min read

  • Bash
  • Linux
  • iptables
  • Reliability

The firewall refresh on a control node does three things in order: apply the new ruleset, prove that an administrator can still get in, and only then confirm the change. Until it is confirmed, a watcher is waiting to put the previous ruleset back. That is the whole safety model — confirmation is earned, not assumed.

One of the proofs is simple: is sshd still listening on port 22?

It passed all morning. Then, at 10:14, the log said this:

[ERROR] verification FAILED: nothing is listening on port 22
[ERROR] NOT confirming - previous ruleset is restored automatically within 300s

sshd had been listening the entire time. The ruleset was fine. The safety net did exactly what it was built to do, and rolled back a perfectly healthy change — because the check that fed it had lied.

The check

In essence, it was this:

set -o pipefail
...
ss -tln | grep -qE ':22[[:space:]]'

List the listening TCP sockets, and succeed if one of them is on port 22. There is nothing wrong with either command. The problem is what happens between them.

What grep -q actually does

The GNU manual is precise about it: with -q, grep will “exit immediately with zero status if any match is found.” Immediately. It does not read the rest of its input — it has its answer, so it leaves.

When grep exits, the read end of the pipe closes. If ss still has output to write, its next write() goes to a pipe that nobody is reading, and the kernel answers with SIGPIPE. ss is killed by the signal. Bash reports a command killed by signal N as exit status 128 + N, and SIGPIPE is 13 — so ss “exits” 141.

Without pipefail, nobody would notice. A pipeline’s status is its last command’s status, grep’s 0, and the check passes.

With pipefail, a pipeline’s status is the last non-zero status in it. That is ss’s 141. The search succeeded; the pipeline failed; the if read false; the log said nothing was listening.

LUCKY — ss finishes writing before grep leaves ss -tln writes everything, exits 0 grep -q ':22' match found, exits 0 statuses 0 · 0 pipefail → 0 sshd listening confirm UNLUCKY — grep leaves while ss is still writing ss -tln writes the first chunk grep -q ':22' matches, exits 0, pipe closes next write() → SIGPIPE ss dies: 128 + 13 = 141 statuses 141 · 0 pipefail → 141 "nothing on :22" roll back Same socket list, same match. Which lane you get depends on scheduling and output size.
The match succeeds in both lanes. In the second, the writer is killed for writing to a pipe nobody is reading any more, and pipefail reports that as the pipeline's result.

Why it passed all morning

Whether ss gets killed depends on timing. If it has written everything before grep finds the match and exits, there is no next write() and no signal. If grep finds the match in an early chunk and leaves while ss still has lines to go, ss dies.

Which of those happens depends on how much output there is, how the writer buffers it, and how the scheduler interleaves two processes. On a quiet host with a short socket list, ss usually wins. Usually is what made this expensive: a check that fails one run in fifty doesn’t get investigated — it gets re-run, and it passes, and everyone moves on.

The first time, it failed silently

This was not new to me. Earlier, while working out why the firewall did not come back after a reboot, an installer I had written contained this:

if systemctl list-unit-files | grep -q '^ipset\.service'; then
    systemctl enable ipset
fi

“If the unit exists, enable it.” Same shape, same bug — but here systemctl has a few hundred lines to print, and the match lands in the middle. It lost the race reliably, so the if read false exactly when the unit existed, and the enable was skipped. No error, no warning, a clean exit from the installer. The unit that restores the firewall’s address sets at boot was simply never enabled.

The two failures pointed in opposite directions. One skipped an action silently. The other fired a safety mechanism for no reason. Both came from the same place: a successful search, reported as a failed pipeline.

That story, and the three other causes stacked on top of it, is in the firewall that didn’t survive a reboot.

The fixes

Capture first, then match. This is what the refresh script does now:

sshd_listening() {
    local out
    if out="$(ss -tln 2>/dev/null)"; then
        grep -qE ':22[[:space:]]' <<< "$out" && return 0
    fi
    ...
    return 1
}

Command substitution reads everything ss writes, so ss can never be cut off. The here-string gives grep its input without a pipe. And there is a bonus: ss’s own exit status is now tested separately, so “ss failed” and “nothing matched” are no longer the same result.

Test existence without a pipe at all. For the installer, the question was never “does this string appear in a listing” — it was “does this unit exist”:

if systemctl cat ipset.service >/dev/null 2>&1; then
    systemctl enable ipset
fi

grep -c is safe; grep -q is not. Counting has to read all of the input, so the writer is never abandoned. The refresh script counts live firewall rules with grep -c in a pipeline, and my handover notes warn against “simplifying” it to -q.

Process substitution also avoids it — grep -q pattern < <(cmd) — because pipefail does not see a process substitution’s status. You lose that status too, though, so capturing is usually better.

What is not a fix: turning pipefail off. That is how the backup check that verified nothing went green while checking nothing: without pipefail, a failed command upstream vanished behind a successful one downstream.

The two bugs are mirror images

SettingWhat went wrong
all-tables-okno pipefailthe writer failed; the pipeline reported success
this postpipefailthe reader succeeded early; the pipeline reported failure

pipefail answers one question: did anything in this pipeline exit non-zero? A reader that stops early makes its writer exit non-zero by design. So neither setting is safe by default. What is safe is not asking one pipeline two questions at once — “did the command work?” and “did the output contain X?” — when its exit status can only answer one of them.

What I took from it

  • Any early-exit reader is a candidate. grep -q, grep -m1, head -1, awk '… {exit}' — under pipefail, every one of them can turn its writer into a 141.
  • A check that fails one run in fifty is not flaky. It is wrong fifty times out of fifty — the other forty-nine just happened not to show it.
  • Read the exit code, not the word “failed”. Exit 141 is a signal, not an error. echo "${PIPESTATUS[@]}" right after the pipeline shows which command produced it, and would have shown 141 0 here in seconds.
  • Safety checks need the most care, not the least. A verification step that can fire spuriously teaches people to override it. The fixed version carries a comment explaining exactly why it is written the long way, so nobody “tidies” it back.

Both of these came out of the firewall automation platform — one in the installer that restores the firewall at boot, one in the check that decides whether a new ruleset is kept.

References

← All posts