Blog

set -e skipped the one line that mattered

An install script hit a harmless error on a CRD I do not even use, and exited. The line it never reached was the one the whole platform depended on — and its absence would have surfaced an hour later as something else entirely.

· 7 min read

  • Bash
  • Kubernetes
  • ArgoCD
  • kubectl
  • Debugging

My ArgoCD install script does four things in order: apply the manifests, wait for the rollout, patch a ConfigMap, restart the repo server. It runs under set -euo pipefail, like every other script in the repo.

It failed on step one:

The CustomResourceDefinition "applicationsets.argoproj.io" is invalid:
metadata.annotations: Too long: may not be more than 262144 bytes

That error is well understood. Client-side kubectl apply records the entire manifest in a last-applied-configuration annotation, and Kubernetes caps an object’s annotations at 256 KiB in total. ArgoCD’s ApplicationSet CRD is bigger than that. The fix is --server-side, which tracks ownership in managedFields instead and has no such limit.

What made this interesting was not the error. It was what the error prevented.

The line that never ran

Here is the shape of the script:

set -euo pipefail

kubectl apply -n argocd -f .../install.yaml        # ← failed here

kubectl -n argocd rollout status deploy/argocd-server

kubectl -n argocd patch configmap argocd-cm --type merge \
  -p '{"data":{"kustomize.buildOptions":"--enable-helm"}}'   # ← never reached

kubectl -n argocd rollout restart deploy/argocd-repo-server

That patch is the single most important line in the repository.

My overlays render a Helm chart through Kustomize. Without --enable-helm, older Kustomize produces no output for those entries and reports no error. ArgoCD then syncs, finds nothing to apply beyond a couple of plain manifests, and reports Synced / Healthy. (Newer Kustomize — v5.7.1 in my testing — at least refuses loudly. ArgoCD ships its own copy, so which behaviour you get depends on the ArgoCD version, not your laptop.)

Green dashboard. Nearly empty namespace.

install-argocd.sh · set -euo pipefail 1 kubectl apply … install.yaml ✗ CRD annotation larger than 262144 bytes — a CRD I do not use set -e → exit 2 rollout status deploy/argocd-server never ran 3 patch argocd-cm --enable-helm never ran — the one line everything depends on 4 rollout restart argocd-repo-server never ran ABOUT AN HOUR LATER kubectl apply root-app.yaml the overlays need --enable-helm ArgoCD: Synced / Healthy with older Kustomize — no error anywhere kubectl get pods → almost nothing just the plain Services from base/ The fix kubectl apply --server-side --force-conflicts The habit assert argocd-cm says --enable-helm
The error was loud and harmless. The consequence was silent and arrived an hour later, two layers away, with nothing connecting the two.

Why this is worse than it sounds

ArgoCD runs perfectly well without ApplicationSet. I do not use it. If the CRD had failed on its own, nothing would have broken.

Instead:

What happenedWhat I would have seen
CRD exceeds the annotation limitA red error, immediately, about a CRD I do not use
set -e aborts the script—
--enable-helm never set—
Root app applied an hour laterSynced / Healthy, and almost no pods

The signature failure of the entire project, arriving an hour after its cause, with nothing connecting the two. I would have gone looking in the overlays.

The repo already knew about the annotation limit, incidentally. The kube-prometheus-stack Application sets ServerSideApply=true for exactly this reason, with a comment explaining why. It had simply never been applied to ArgoCD’s own installation.

The fix, and the better habit

kubectl apply -n argocd --server-side --force-conflicts -f .../install.yaml

--force-conflicts lets it reconcile a namespace where an earlier client-side apply had already claimed fields.

Terminal: ArgoCD services, deployments, statefulset and network policies serverside-applied; argocd-cm patched to enable the Helm renderer in Kustomize; the repo server restarted; the initial admin password line is redacted; ArgoCD is up.
The same script after the fix. Every object serverside-applied, then configmap/argocd-cm patched and the repo server restarted — the two steps that never ran the first time. The blurred box is the bootstrap admin password.

But the durable fix is not that flag. It is noticing that set -e converts an error into an absence, and an absence is much harder to see than an error.

Two things I now do in bootstrap scripts:

Verify the critical settings at the end, not just the exit code.

kubectl -n argocd get cm argocd-cm \
  -o jsonpath='{.data.kustomize\.buildOptions}'
# must print: --enable-helm

One line, and it turns “the script exited 0” into “the thing the script existed to do is actually true”.

Put the load-bearing line early, or isolate it. If a step matters more than the steps around it, do not let an unrelated failure upstream decide whether it runs.

A second one, same shape

The same build had a CI job fail twice in a row with two different errors, and the first one hid the second.

Attempt one: both matrix legs failed in under twelve seconds with

buildx failed with: Learn more at https://docs.docker.com/go/build-cache-backends/

GitHub truncated the real message. In full it was “Cache export is not supported for the docker driver” — the workflow used cache-to: type=gha without a docker/setup-buildx-action step, so it was running on the default builder, which cannot export a cache.

Fixed that. Attempt two failed with something much clearer:

invalid tag "ghcr.io/Syed-Amjad/web:5446e1e…": repository name must be lowercase

github.repository_owner returns the owner with its real capitalisation, and Docker repository names must be lowercase. GitHub Actions expressions have no lowercase function, so it is done in bash: ${GITHUB_REPOSITORY_OWNER,,}. The second error was only reachable once the first was out of the way.

The useful signal there was the timing. Two matrix legs with different dependencies, failing identically in twelve seconds, is not a code problem — that is less time than pulling a base image. Identical fast failures across a matrix mean configuration.

What I took from it

  • set -e turns a cosmetic error into a silent omission further down the file. The thing that breaks is not always the thing that errors.
  • Assert the outcome, not the exit code. A script that finished is not a script that worked. One jsonpath check is cheaper than an hour.
  • Errors queue up. Fixing one does not mean you are done; it means you can now see the next one. Expect to iterate rather than assuming a single cause.
  • Identical fast failures across a matrix are configuration, not code. Different inputs failing the same way in the same few seconds narrows it immediately.
  • When a repo already documents a trap somewhere, check everywhere it applies. The annotation limit was known, commented and handled — in one file out of two.

From the GitOps delivery platform build.

References

← All posts