Bootstrapping Terraform

I would say at this point I'd consider myself a Terraform pro™️. This is my preferred way of bootstrapping Terraform!

A place for the Terraform state file

Terraform code is essentially a custom language for a JSON file. This JSON file holds data that maps onto our cloud infrastructure, and Terraform helps us keep these in sync. Each operation we perform against a Terraform project reads this state to determine which resources have changed.

Often times, we have private or semi-private identifiers such as account identifiers, ephemeral passwords, or identifiers of keys. Information that is nice to keep out of our Git history. What I'm trying to say is that we want to store the state in a secure place.

Terraform lets us configure where to store this state file. And I think most users are already on a hyperscaler so it only makes sense to store the state on said hyperscaler, typically in an object storage such as S3. If hyperscalers are not your thing there are still plenty of other options :) This also happens to solve our authorization problem. Lock down access to the S3 bucket and now you've also guarded the state from unwanted onlookers.

Now here is the problem... If I want to put my Terraform state file on S3, wouldn't it also be nice to define the S3 bucket using Terraform? Sounds like a chicken-and-egg problem doesn't it?

Let's use multiple projects

To solve our problem, we simply introduce another Terraform project. The sole purpose of this new Terraform project is to provision the AWS S3 bucket (and optionally DynamoDB table for state locking) so that the main project can be configured using these resources as the state backend. I typically put this inside a folder named bootstrap with a short README document on why it exists.

bootstrap/
  .terraform.lock.hcl
  bootstrap.tf
  bootstrap.tfstate
  README.Md
infra/
  .terraform.lock.hcl
  terraform.tf
  ...

The bootstrap project uses Terraform's local file backend. I know I said I would like to keep this out of Git earlier, but I think this file is pretty harmless to keep in Git as it's mostly just config for one bucket and maybe one DynamoDB table.

If all goes well, you now have all infrastructure declared via Terraform! Happy hacking!

Side note: this simple approach makes it really easy to destroy all the resources associated in the project if you go bankrupt and have to shut down too :)