← All posts
Devesh SainiDevesh Saini — SRE · Cloud & DevOps Engineer

Terraform state: lessons from three disasters

Terraform state is the most dangerous file in your infrastructure, and most teams treat it more casually than their application code.

Disaster one: two engineers ran apply concurrently against local state. The fix is table stakes now — remote state with locking from day one — but the deeper lesson was that convenience defaults are production risks.

Disaster two: a CI misconfiguration pointed a destroy at the wrong workspace. Lesson: production applies go through a plan that a human reads, environments live in separate state files with separate credentials, and destroy requires a flag nobody sets by accident.

Disaster three: a stuck lock, a hasty force-unlock, and a state file that no longer matched reality. Lesson: state surgery is an incident. Declare it, snapshot the state first, and do imports and moves one resource at a time with a second pair of eyes.

Boring process is what makes infrastructure as code safe. The code is the easy part.

// more like this

AWS Cloud Explained: Services, Benefits & How to Get Started

Blameless postmortems that actually change things

Kubernetes Requests and Limits: A Practical Sizing Guide

Comments

Comments are moderated before appearing. Leave your email to be notified when yours is approved or replied to.