A high-traffic secondary ticketing platform on AWS EKS — engineered to absorb the traffic spike of a major on-sale event without dropping orders, and to be fully reproducible as code.
99% of the traffic is ordinary browsing. The other 1% is a panic: a major artist announces a tour, and tens of thousands of concurrent users hammer the inventory API inside the same 60-second window. The platform has to absorb that spike without dropping orders — every dropped order is lost revenue — while not paying for that peak capacity 24/7.
I owned the infrastructure end to end: the EKS cluster in Terraform, the Helm charts for each service, the autoscaling behavior, and the full observability layer.
Traffic enters through an AWS Load Balancer Controller–backed Gateway, hitting four EKS-hosted services. Terraform manages everything below the application; Prometheus watches everything above it.
Solid arrows = request path. Dashed = metrics scrape. Dashed box = everything provisioned as code.
Load tests showed the inventory service falling over at 8× normal RPS. A CPU-based autoscaler couldn't help — the service goes from idle to 10,000 RPS in about 90 seconds, well before CPU registers the wave.
http_requests_per_second.PodDisruptionBudgets so a Spot drain mid-event couldn't take all replicas down at once.Peaked at 14× baseline. Inventory scaled 3 → 22 pods in under 2 minutes, p99 latency held under 400ms, zero dropped orders.
The cluster and all supporting infrastructure lived in CloudFormation stacks that had accumulated a year of console-clicked drift — and nobody had a complete map of what each stack owned. The goal: move everything into Terraform without recreating the cluster, and without a window where both systems managed the same resource.
terraform imported each resource into state.terraform plan was a clean no-op — proof the code matched reality — before touching anything.A CFN security group was referenced by three other stacks via ImportValue. I migrated the consuming stacks to Terraform data sources first, so the exporting stack had no dependents and could be deleted cleanly. Map cross-stack dependencies before you start, not as you hit them.
Full cluster infra — VPC, subnets, security groups, IAM/IRSA roles, the EKS cluster, and node groups — migrated with zero downtime and zero resource replacements. Every change since goes through a reviewable terraform plan in a PR.
The old alerts fired on raw error rate — one error at 2am read as "100% error rate" and paged; meanwhile slow regressions that stayed just under the threshold during busy hours went unnoticed.
PrometheusRule manifests kept inside each Helm chart — versioned, deployed, and rolled back with the service. No click-ops in Grafana.On-call pages dropped ~70% in the first month. The next real incident — a pricing regression sending 0.8% of requests to 500 — tripped the fast-burn alert in 4 minutes, ~40 minutes earlier than the old threshold would have.