Blog

The canary header that never arrived

Header-based routing is described as free with Istio. It is free for one hop. On a two-hop path it is an application contract, and mine did not implement it — which the measurements showed clearly, once I changed one variable and watched what moved.

· 7 min read

  • Istio
  • Canary
  • Observability
  • FastAPI
  • Distributed systems

My canary setup has two mechanisms. A weighted split for general traffic, and a header override so I can exercise the new version specifically, before anyone else sees it:

http:
  - name: canary-by-header
    match:
      - headers:
          x-canary:
            exact: "true"
    route:
      - destination: { host: api…, subset: v2 }

  - name: weighted-split
    route:
      - destination: { host: api…, subset: v1 }
        weight: 90
      - destination: { host: api…, subset: v2 }
        weight: 10

The header rule is first. First match wins. Send x-canary: true and you get v2, every time.

The weighted split measured beautifully:

Weights in gitRequestsv1v2
90 / 10200171 (85.5%)29 (14.5%)
50 / 50200102 (51.0%)98 (49.0%)
Terminal: demo-canary.sh sends 200 requests; responses by api version are v1 171 (85.5%) and v2 29 (14.5%), against configured weights of 90 and 10.
Counted from the version each response reported — not read back from the VirtualService.

Then I ran the header test.

CANARY_HEADER=1 ./scripts/demo-canary.sh 50
==> Sending x-canary: true — expecting 100% v2

    v1   46   92.0%
    v2    4    8.0%
Terminal: CANARY_HEADER=1 demo-canary.sh 50 sends x-canary true, expecting 100% v2; responses are v1 46 (92%) and v2 4 (8%), against configured weights of 90 and 10.
Expected 0 / 50. Got 46 / 4.

The runbook sent me to the wrong file

My own runbook has a troubleshooting entry for exactly this:

Both versions answer when you sent the canary header Rule order. The x-canary match must be the first entry under http:.

It is first. I checked three times.

What actually gave it away was running the test again after changing the weights:

Weights at the timeHeader test result
90 / 1046 / 4 — 92% / 8%
50 / 5027 / 23 — 54% / 46%

The “header” result tracks the weights.

If the header rule were matching, the weights would be irrelevant. They were not irrelevant. So the header rule was never being evaluated against a request that carried the header.

Where the header dies

curl x-canary: true ingress gateway header present ✓ web present ✓ … then dropped new request api VirtualService 1 · match x-canary → v2 no header 2 · weighted split falls through → v1 90 · v2 10 web/main.py — the call to api async def index(): # no Request parameter r = await client.get(f"{API_URL}/data") # no headers forwarded The mesh routes on the request it sees. This one was built fresh, so it carries nothing from the original.
Every component Istio owns behaved correctly. The header survived the gateway and reached web — and died there, because web makes its own request to api.

The cause is a few lines of application code:

@app.get("/")
async def index():                              # no Request parameter
    async with httpx.AsyncClient(timeout=5.0) as client:
        r = await client.get(f"{API_URL}/data") # no headers forwarded
    …

index() never asks for the inbound request, so it cannot read the header. And the call it makes to api is a brand-new HTTP request that carries nothing from the original.

Istio did everything right. The gateway received the header. The api VirtualService was correctly ordered. There was simply no header left by the time it mattered.

The fix is an application contract

PROPAGATE = {"x-canary", "x-request-id",
             "x-b3-traceid", "x-b3-spanid", "x-b3-sampled"}

@app.get("/")
async def index(request: Request):
    fwd = {k: v for k, v in request.headers.items()
           if k.lower() in PROPAGATE}
    r = await client.get(f"{API_URL}/data", headers=fwd)

That list is not arbitrary. It is the same set distributed tracing needs, for the same reason — Istio’s own documentation is blunt that applications “must propagate this information in HTTP headers”: a service mesh can route on context it can see, but only the application can carry that context across a call it originates.

Anyone who has wired up tracing has written this function. I had not connected it to canary routing, because header routing is usually demonstrated on a single service, where the question never comes up.

I left it broken

I did not ship the fix. The project is published with this demonstration marked as failed, the measurements showing 46/4 where they should show 0/50, and the reason written out.

Two reasons. The honest one is that the cluster had been torn down, and the fix needs a code change, a CI run and a redeploy. The better one is that a published failure with a diagnosis is worth more than a green tick — and this particular failure is the most transferable thing in the whole project. The weighted split working proves Istio was installed correctly. The header not working proves something about how distributed systems actually behave.

It also sits exactly on the line the platform is built around: deploying and releasing are different decisions.

DEPLOY — AUTOMATIC push to main CI builds and pushes the image CI writes the tag → values-api-v2.yaml ArgoCD rolls out the v2 pods no user traffic reaches them RELEASE — DELIBERATE a human edits istio/api-traffic.yaml weights 90 / 10 → 50 / 50 commit and push — reviewed, revertable ArgoCD applies the new weights traffic starts flowing to v2 — then re-measure Broken code can sit in the cluster all day harming nobody. Shipping and releasing are two different decisions.
The header fix is a deploy — code, CI, a new image. Turning traffic onto it would be a separate release. Leaving the failure published rather than rushing a patch keeps those two honest.

What I took from it

  • Change one variable and watch what moves. The header result tracking the weights was the entire diagnosis. One extra run at different weights beat three readings of a correct config file.
  • Your own runbook can be confidently wrong. Mine named a real cause for these symptoms — just not this one. A troubleshooting list is a list of some causes, and treating it as exhaustive sends you to inspect correct files repeatedly.
  • “Free with Istio” usually means free for one hop. Header routing, tracing and anything else carried in request context all need the application to participate the moment there is a second service.
  • Mesh config being correct does not mean mesh behaviour is correct. istioctl analyze was clean. The rule was first. Everything the mesh owned was right, and the feature still did not work.
  • Publish the failure. The expected-versus-measured gap, with the cause, is more convincing than another passing test.

From the GitOps delivery platform build — seventeen defects found, sixteen fixed, this one left standing.

References

← All posts