Zero-Trust Secure Supply Chain Platform

Lab build on a single t3.large in a personal AWS account, provisioned by Terraform and destroyed between sessions. Every error message quoted on this page is copied from the attack transcripts in the repository. The addresses shown are private, inside a VPC that no longer exists.

At a glance

The claim this project had to earn

Most security write-ups show a configuration and assert that it works. This one assumes the opposite until proven: a policy that is accepted by the API server is not a policy that is enforced, and the only way to tell the difference is to attack it and read which component answers.

So the platform was built, and then seven attacks were run against it. Each attack targets one named layer. Each result records the actual error message, because the message is the evidence — it names which control spoke. “Rejected” in a results column could have come from any of four places.

Six of the seven held. One did not, and the reason it did not is the most useful thing on this page.

The second namespace is a test fixture with Pod Security Admission deliberately turned off, and its annotation says so. Without it the Kyverno rule is untestable: in every governed namespace PSA refuses the pod first, so the Kyverno policy would appear to pass while never having been exercised.

Architecture

The design argument is that these are independent, not redundant. Attack 2, above, is the proof.

What was measured

Controls verified before any attack ran

03-verify-enforcement.sh tests behaviour, not YAML. It opens a plaintext connection from outside the mesh, submits a privileged pod, counts the loaded policies, and reads the mesh’s outbound mode.

================ RESULT ================
  passed 15   failed 0   skipped 0

The check that matters most is the first one:

CHECK 1 of 6 — Istio traffic redirection (the silent one)
      BLOCKED ConnectionResetError [Errno 104] Connection reset by peer
  PASS  Plaintext from outside the mesh was REFUSED - redirection works

That connection reset is the only honest proof that traffic is actually passing through the sidecars. Every mesh guarantee is enforced by an Envoy that traffic has to traverse — not by the policy objects, which apply cleanly whether or not anything reads them.

The seven attacks

Audit before Enforce, run as two passes

Every Kyverno policy ships with validationFailureAction: Audit. The attacks were run twice — once in Audit, once in Enforce — and both transcripts kept, because the pair is the demonstration:

The switch to Enforce is a commit, not a kubectl patch — see the defects below for why that distinction turned out to be load-bearing.

Signature identity, verified by hand

The strongest claim here is that compromising the registry alone is not enough. A passing admission check cannot distinguish “the signature matched the pinned identity” from “my policy is looser than I think it is”, so cosign was run twice against the same digest — once with the correct identity, once with a wrong one from the same repository and branch.

The one that failed, and why it is the most valuable result

Attack 6 scales a Deployment to 7 replicas by hand and watches for ArgoCD to revert it. Sixty seconds later it was still at 7.

ArgoCD v2.13.2 did not recognise .status.terminatingReplicas, a Deployment status field present in the Kubernetes the cluster was running (k3s v1.36.5). Its structured-merge diff could not compute a difference. With no computed difference, selfHeal had nothing to correct — and never ran.

Everything on the dashboard said this was fine. The Application reported Healthy. syncPolicy.automated.selfHeal reported true. The only visible symptom was a sync status of Unknown, which reads like a transient.

Pinning the GitOps controller to a version older than the cluster silently disabled drift correction. Nothing alerted. The control that is supposed to notice unauthorised change had itself stopped working, and the mechanism that would report that is the same mechanism that was broken.

That is worth more than a seventh passing row. The whole premise of this project is that a control which reads as enabled may not be enabled; attack 6 is that premise happening to the project itself.

Defects found

Fifteen defects surfaced. Six were in the platform or its dependencies; nine were in my own tooling, and those are listed too, because a results table containing only the expected outcomes tells a reader the interesting rows were deleted.

In the platform and its dependencies

In my own tooling — the honest half

Nine defects were mine, and they share one shape: a check that looked right and tested the wrong thing.

Every one of these is now fixed, and each fix was verified by reproducing the failure first: a 0700 directory for the glob, a stubbed kubectl returning no nodes, a simulated chained conflist, five fake k3s layouts for the path discovery, six helm release states, and a fake state file in the wrong account.

Defect 15 is the one I would want a reader to take away. A teardown script that prints the account rather than checking it is worse than no teardown script, because it is trusted. It now reads the owning account out of the state file’s resource ARNs, compares it to the caller, and refuses on a mismatch.

What this platform does not claim

These limitations are in the repository too.

• Falco detects; it does not prevent. Attack 7 succeeded by design. In this lab the alert goes to a pod log and falcosidekick. An alert nobody is paged for is the same class of problem as a policy left in Audit.
• Egress filtering is hostname-level, not path-level. For a TLS destination the mesh matches on SNI and does not terminate the connection. Path-level filtering of an external HTTPS call would mean intercepting your own TLS, with the certificate-distribution problem that brings. Hostname-from-a-named-workload is the honest description — and it still defeats the attack: exfiltration needs a destination, and an attacker cannot add one without write access to the repository.
• selfHeal reverts resources ArgoCD manages. A pod created by hand is not drift — it is unmanaged, and reconciliation has no opinion about it. The layer that stops the hand-created pod is admission control, not GitOps. orphanedResources.warn is set for that reason, and a warning in a UI is weaker than a rejection.
• An allowlist is a name check, not an identity check. Anyone who can push to ghcr.io/syed-amjad satisfies the registry policy. Closing that gap is what the signature policy is for. Neither is sufficient alone, which is the argument for having both.
• Single node. Everything runs on one t3.large at about $0.083 an hour. That is enough to demonstrate every control and not enough to say anything about behaviour under scale or node failure.

Stack and cost

Total AWS spend for the build: one t3.large for roughly six hours, plus a 40 GB gp3 root volume — under $1. The instance is destroyed between sessions, and the whole platform rebuilds from git in about an hour.

Repositories

CI has no kubeconfig and no cluster credentials. A compromised application pipeline can push a bad image; it cannot rewrite cluster configuration directly — and Kyverno then refuses the bad image at admission, which is the point of the split.

Every quoted message on this page is in one of the two transcripts, in context — including the ones that record my own mistakes. The results were filled in from the transcript while it was in front of me, not written from memory afterwards.

Relationship to my production work

This is the same default-deny with an explicit allowlist architecture I ran in production for host firewalls across a Linux fleet — applied here at seven layers, from the CI pipeline down to the kernel.

The firewall enforces at L3/L4 on source address and port. Istio enforces at L7 on cryptographically verified workload identity. Same design, different layer — and the same lesson keeps arriving from both directions: the most dangerous control is the one that reports healthy while doing nothing.

The build, start to finish

Expand the walkthrough — offline tests to the Audit pass

1 · Policies tested before any cluster exists

2 · The first verification, in Audit

3 · The attacks in Audit — admitted and recorded