Projects

Production Kubernetes Platform on AWS

A kubeadm cluster built from scratch on a custom multi-AZ VPC — ALB ingress with host-based routing, Helm-packaged services, autoscaling validated under load, and full observability.

Kubernetes · kubeadm · AWS · Helm · Terraform · Prometheus

Production

Scope

An end-to-end Kubernetes platform on AWS, built without a managed control plane — every component provisioned, configured and debugged by hand, then packaged for repeatability.

The point was not to get a cluster running. It was to understand every layer well enough to diagnose it when it broke, which it did, repeatedly.

Internet DNS + ACM TLS, custom domain VPC — 2 availability zones Application Load Balancer AWS Load Balancer Controller · host-based routing us-east-1a public subnet · NAT gateway private subnet control plane (kubeadm) worker · containerd us-east-1b public subnet · NAT gateway private subnet worker · containerd worker · containerd Flannel CNI — pod network across both zones ECR container images Helm backend + PostgreSQL Observability Prometheus Grafana HPA metrics Horizontal Pod Autoscaler — scaling behaviour validated under generated load, not just configured and assumed to work.
Multi-AZ VPC, kubeadm control plane, ALB ingress via the AWS Load Balancer Controller, Flannel pod network.

What it covers

Network foundation. A custom VPC across two availability zones, with public and private subnets, NAT gateways for outbound traffic from private nodes, and security groups scoped per tier. Nodes sit in private subnets; nothing in the cluster is directly reachable from the internet.

Cluster. kubeadm-provisioned control plane with containerd as the runtime and Flannel for the pod network — no managed control plane, so certificate handling, component flags and cluster bootstrapping were all explicit.

Ingress. AWS Load Balancer Controller provisioning an ALB from Ingress resources, with host-based routing to different services and ACM certificates for TLS on a custom domain.

Applications. Multi-service deployment packaged as Helm charts — an application backend plus PostgreSQL — with images stored in ECR.

Observability and scaling. Prometheus and Grafana with Kubernetes dashboards, and a Horizontal Pod Autoscaler whose behaviour was verified by generating load rather than assumed from the manifest.

The cluster, running

kubectl get nodes -o wide showing a control plane and two workers on Kubernetes v1.29.15 with containerd, followed by kubectl get pods -A listing flannel, CoreDNS, etcd, apiserver, scheduler, metrics-server, the AWS Load Balancer Controller and application pods.
Three nodes on v1.29.15 with containerd, and every control-plane component visible as a pod — etcd, apiserver, controller-manager, scheduler — because nothing here is managed for me.
helm install of the aws-load-balancer-controller chart into kube-system, followed by kubectl get pods showing two controller replicas running.
AWS Load Balancer Controller installed via Helm and bound to the cluster's VPC — this is what turns an Ingress resource into a real ALB.
A loop of twenty curl requests through the ALB hostname with a Host header, all returning status ok, alongside kubectl get ingress showing the ALB address and pods distributed across worker nodes.
Twenty requests through the ALB with a Host header, all returning {"status":"ok"} — host-based routing resolving to pods spread across both worker nodes.
Browser showing the application health endpoint returning a JSON status ok response.
The same endpoint from a browser. Unglamorous, and the only thing that actually proves the whole path works.

What actually took the time

The build was straightforward. The debugging was the education:

  • ImagePullBackOff — ECR authentication and node IAM permissions, not the image
  • ALB 504 gateway timeouts — target group health checks pointing at a path the application didn’t serve
  • Ingress routing not matching — host rules and ingress class resolution
  • TLS certificate mismatch — certificate domain against the hostname actually being requested
  • DNS resolution — record propagation and the gap between what the ALB advertised and what the domain resolved to

Each of those failed in a way where the symptom pointed somewhere other than the cause. That is the part of running Kubernetes that no tutorial teaches, and the reason I built the cluster the hard way rather than clicking through a managed service.

Why kubeadm rather than EKS

For production I would generally take the managed control plane. For learning, EKS hides exactly the parts worth understanding — etcd, API server configuration, certificate rotation, node bootstrapping. Building it by hand meant that when something broke I had to know which component owned the problem.

It is also the substrate the CKA exam tests, which made it double as preparation.