Blog

Notes from building and breaking infrastructure — mostly about the failures, because those are the parts worth writing down.

  • · 7 min read

    One allowlist entry short, four applications broken

    Four ArgoCD Applications failed, and only one of the four errors named the real cause. The others pointed confidently at the wrong layer — and the eventual fix meant admitting that a security control I had written could not survive contact with the thing it was protecting.

    • ArgoCD
    • GitOps
    • Kubernetes
    • RBAC
    • Debugging

  • · 7 min read

    The canary header that never arrived

    Header-based routing is described as free with Istio. It is free for one hop. On a two-hop path it is an application contract, and mine did not implement it — which the measurements showed clearly, once I changed one variable and watched what moved.

    • Istio
    • Canary
    • Observability
    • FastAPI
    • Distributed systems

  • · 8 min read

    STRICT mTLS and Prometheus cannot both be right

    Every scrape target sat DOWN with "connection reset by peer" while the endpoint answered 200 from inside the pod. Nothing was broken — the mesh was enforcing exactly what it had been told to. It took four attempts to fix it without giving up the control, and the fix broke something two files away.

    • Istio
    • Prometheus
    • mTLS
    • Kubernetes
    • Observability

  • · 7 min read

    set -e skipped the one line that mattered

    An install script hit a harmless error on a CRD I do not even use, and exited. The line it never reached was the one the whole platform depended on — and its absence would have surfaced an hour later as something else entirely.

    • Bash
    • Kubernetes
    • ArgoCD
    • kubectl
    • Debugging

  • · 7 min read

    terraform apply succeeded. The box had no way out.

    A default VPC with no subnets, no internet gateway, and a default route still pointing at the gateway that used to exist — in state blackhole, which AWS does not consider an error. Then the same stack failed again, on an apostrophe, after it had already built the instance.

    • Terraform
    • AWS
    • VPC
    • Networking
    • Debugging

  • · 8 min read

    Five ways to verify a fix without testing it

    A passing check, a clean query, a plausible error, a success message. Each of these looked like evidence, and each was measuring something other than the change I thought I was testing. The habits I use now so that "verified" means something.

    • Debugging
    • Python
    • MySQL
    • Linux
    • Testing

  • · 9 min read

    What sudo actually matches

    A sudoers rule reads like a list of allowed commands. sudo reads it as a pattern over one joined string, with a literal interpreter path, a scrubbed environment, and complete trust in the filesystem. Five things that mattered when locking down scripts the web tier runs as root.

    • Linux
    • sudo
    • Security
    • Bash
    • Least privilege

  • · 9 min read

    HTTP 405 was the healthy answer

    "Checks are failing" is not a diagnosis. Two fleet sweeps turned it into one — including a probe where Method Not Allowed meant everything was fine. The biggest cause of failed checks was not in the checking code at all.

    • HTTP
    • Python
    • FastAPI
    • Observability
    • Debugging