Blog

terraform apply succeeded. The box had no way out.

A default VPC with no subnets, no internet gateway, and a default route still pointing at the gateway that used to exist — in state blackhole, which AWS does not consider an error. Then the same stack failed again, on an apostrophe, after it had already built the instance.

· 7 min read

  • Terraform
  • AWS
  • VPC
  • Networking
  • Debugging

I was building a Kubernetes delivery platform on a single EC2 box, and the archive I started from had no provisioning at all. The bootstrap script’s second line said “Run once on a fresh t3.xlarge” and left the box as an exercise for the reader.

So I wrote a small Terraform stack. One instance, one security group, one key pair, the default VPC. The obvious thing.

The plan failed before it had planned anything:

Error: Invalid index
  on compute.tf line 13, in resource "aws_instance" "node":
  13:   subnet_id = data.aws_subnets.default.ids[0]
    │ data.aws_subnets.default.ids is empty list of string

The given key does not identify an element in this collection value:
the collection has no elements.

The default VPC was not what I assumed

A default VPC comes with a subnet in every availability zone and an internet gateway attached. That is the entire point of it. Mine had neither.

aws ec2 describe-subnets --region us-east-1 --query 'Subnets[].SubnetId'
# (empty)

aws ec2 describe-internet-gateways --region us-east-1
# { "InternetGateways": [] }

Someone — me, at some point, cleaning up a lab — had deleted them. What was left was the VPC shell, and this:

aws ec2 describe-route-tables --region us-east-1 \
  --filters "Name=vpc-id,Values=vpc-0…" \
  --query 'RouteTables[].Routes'
[
  { "DestinationCidrBlock": "172.31.0.0/16", "GatewayId": "local",
    "State": "active" },
  { "DestinationCidrBlock": "0.0.0.0/0",     "GatewayId": "igw-0…",
    "State": "blackhole" }
]

A default route pointing at an internet gateway that no longer exists.

blackhole is not an error state

This is the part worth sitting with. AWS does not consider that route table broken. Its API defines blackhole as a route whose “target isn’t available” — a state, not a fault. The table is served, and it is attached to anything that asks for it. The route is present. It simply drops traffic.

The empty subnet list was the lucky break: it failed loudly, at plan time. The obvious next move — create a subnet and let it inherit the VPC’s main route table — would have gone like this:

StepResult
terraform apply✅ succeeds
Instance boots✅ running, passes both status checks
ssh ubuntu@<ip>❌ times out

Status checks pass because they never leave AWS’s own network. SSH cannot work at all: with no gateway attached there is nothing for a packet from the internet to enter through — and even if there were, the reply would follow the default route straight into the blackhole. A default route carries your replies, not just your outbound traffic.

And an SSH timeout has a very familiar list of suspects: the security group, the /32 it allows (has my home address changed?), the key, a host firewall. The route table of a default VPC is not on that list, because nobody edits it. I would have spent an evening re-checking a security group that was correct.

INHERITED — WHAT THE ACCOUNT HAD internet internet gateway deleted — nothing to enter through default VPC main route table 172.31.0.0/16 → local active 0.0.0.0/0 → igw-0… (gone) blackhole new subnet — inherits the main table EC2 instance running · status checks pass unreachable · cannot reach out DECLARED — WHAT THE STACK OWNS internet aws_internet_gateway same VPC shell aws_route_table — the stack's own 0.0.0.0/0 → aws_internet_gateway active the main table is left alone — it is not this stack's 172.31.200.0/24 · explicit association EC2 instance SSH in from a /32 packages and images out
Inheriting account state meant inheriting a route nobody was maintaining. Declaring four small resources means the stack no longer depends on it — and terraform destroy removes every part.

The fix was to stop inheriting

Four small resources instead of a data source:

resource "aws_internet_gateway" "igw" {
  vpc_id = data.aws_vpc.default.id
}

resource "aws_subnet" "node" {
  vpc_id                  = data.aws_vpc.default.id
  cidr_block              = "172.31.200.0/24"
  map_public_ip_on_launch = true
}

resource "aws_route_table" "public" {
  vpc_id = data.aws_vpc.default.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
}

resource "aws_route_table_association" "node" { … }

I deliberately did not repair the VPC’s main route table. That table belongs to the account, not to this stack, and quietly rewriting it would be a side effect reaching outside what the stack claims to own. The subnet gets its own table and an explicit association.

The instance then came up on 172.31.200.184 — an address from the subnet the code declared, not one discovered from account state nobody was maintaining.

The other half: a clean plan is not a promise

The same stack failed again later, differently, and for a reason worth recording.

Error: creating VPC Security Group Rule
  InvalidParameterValue: Invalid rule description. Valid descriptions are
  strings less than 256 characters from the following set:
  a-zA-Z0-9. _-:/()#,@[]+=&;{}!$*

My description read "SSH from the operator's address only". The apostrophe is not in that set.

terraform plan had been perfectly happy — 9 to add, 0 to change, 0 to destroy. It validates types and references, not string contents against service-side rules.

And because Terraform creates in dependency order, and the instance does not depend on the ingress rule, it built the instance first. The apply then failed on the rule.

The result was a running, billing t3.xlarge with no open SSH port — the one state where you are paying full price for a machine you cannot reach, arrived at via a validation error about punctuation.

The state file stayed correct throughout, which is why the fix was a one-line edit and a re-apply: 1 to add, 0 to change, 0 to destroy. That is the argument for IaC in a sentence. The same failure click-opsed leaves you reconciling a console by hand.

What I took from it

  • “It applied successfully” and “it works” are different claims. A clean plan is a promise about intent, not about what the API will accept — and certainly not about whether packets move.
  • A loud failure is a gift. The empty-list error cost me ten minutes. The blackhole route would have cost an evening, because its symptom points at the security group rather than at itself.
  • Do not inherit infrastructure you did not declare. Data sources are convenient right up to the moment the thing they look up has been quietly deleted. If the stack depends on it, declare it.
  • A partial apply is a normal outcome, not a corrupt one. Read the state, fix the one line, re-apply. Panic is not a step.
  • Check the thing you did not change. The route table was not part of my work, had no reason to be wrong, and was the whole problem.

This came out of building a GitOps delivery platform — seventeen defects, all written up.

References

← All posts