Automated Infrastructure Management with Terraform Backend and S3 Bucket Integration

Introduction

When creating infrastructure as code (IaC) via Terraform, it is necessary to maintain a history of the infrastructure being built. It accomplishes this using a state file, which is merely a straightforward JSON file that outlines the infrastructure that Terraform has built and any modifications that might be made. This file should be checked into git or any other source code management system that is being used because, by default, it is typically kept on the local machine where Terraform commands are executed and we all know that it is not ‘if’ my local machine goes down, but ‘when’.

Prerequisites

Terraform installed
AWS billing account
AWS CLI (see here for CLI installations)
AWS IAM account with necessary permissions

Create S3 Bucket with Versioning

When using a S3 bucket as a backend, it improves collaboration as multiple team members can modify the state file without causing any infrastructure/git conflicts.

Via AWS CLI, you can create a s3 bucket by:

aws s3 mb s3://myTFbackend

Once the bucket is created, enable versioning so that you can revert back to previous state files if needed:

aws s3api put-bucket-versioning --bucket myTFbackend --versioning-configuration Status=Enabled

Create DynamoDB Table

To enable state locking and preventing write operations to your state file while another write operation is running (when 2 or more people attempt to modify the state file at the same time), we will use a Dynamo Table for state locking and consistency.

aws dynamodb create-table --table-name tfStateLock --attribute-definitions AttributeName=LockID,AttributeType=S --key-schema AttributeName=LockID,KeyType=HASH --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

Configure Terraform Backend

Now that the stage is set with a S3 bucket to hold the statefile and a DynamoDB table to take care of locks, we can modify the configuration file (e.g. main.tf) in Terraform to use these objects:

terraform {
  backend "s3" {
    bucket         = "myTFbackend "
    key            = "terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "tfStateLock "
  }
}

Initialize Terraform

Even though you may have already initialized your environment before attempting to switch the state backend to S3, whenever there is a change in configuration of the backend then it is recommended to re-initialize so that the state file will be sent to the S3 bucket:

terraform init