Event-Driven Automation: Creating a Text File upon EC2 Instance Lifecycle Events

TL;DR

Build an event-driven automation pipeline using EventBridge, SNS, and Lambda to capture EC2 lifecycle events and write them as JSON files to S3. This Terraform-based solution demonstrates how to create serverless triggers that respond to infrastructure changes automatically.

Introduction

In the world of cloud computing, Amazon Elastic Compute Cloud (EC2) instances are the backbone of countless applications and services. As the need for automation and orchestration grows, it becomes essential to use the capabilities offered by EC2 instance lifecycle events. These events, triggered during various stages of an instance's lifecycle, provide a powerful mechanism to perform custom actions and streamline operations.

In this blog post, we will explore how to use the potential of EC2 instance lifecycle events by focusing on a specific use case: creating a text file to a S3 bucket upon the occurrence of a lifecycle event. We will look at the underlying concepts, walk through the necessary setup, and demonstrate step-by-step how to achieve this automation using AWS services.

By the end of this tutorial, you will have a solid understanding of creating an event driven automated trigger that can be used to set an action. In this simple example, we will use EC2 instance lifecycle events at a trigger with the action of writting key fields of this transaction to a text file to S3.

In the software development theme, we will somewhat use the Don't Repeat Yourself principle and create the below in Terraform.

Logical Flow

Tasks

Create S3
Create Lambda
Modify Lambda permission to write to S3
Create SNS Topic
Create SNS subscription to Lambda
Create Eventbridge rule to trigger action to SNS
Create an EC2 instance and confirm successfull workflow

S3 Creation

We will create a S3 bucket with a random 16 alphanumeric lowercase character string with 'events' as the suffix.

I used the random_string resource (16 chars, lowercase, no special chars) to build a unique bucket name with an -events suffix, then created an aws_s3_bucket referencing that random string.

Lambda

Since Lambda does require some type of code, i will be using Python to create a json file based on events and send that json file to the S3 bucket. I will start by zipping up the Python file and then use that zip file to upload to the Lambda function that i will create.

Warning

Make sure to update the BUCKET_NAME variable in your Python code to match your actual S3 bucket name. Using a hardcoded bucket name will cause the Lambda to fail if the bucket doesn't exist.

Python Code To Be Saved As a File

The Lambda is short: it imports boto3, takes the incoming event, builds a filename from context.aws_request_id, and calls s3.put_object with the JSON-dumped event body. Returns a 200 with the saved path.

Pro Tip

Consider using environment variables for the bucket name instead of hardcoding it. This makes your Lambda more portable and easier to configure across different environments.

Zip Python File

I used Terraform's archive_file data source to zip lambda_function.py into lambda_function.zip at plan time.

Lambda Function Creation

The Lambda Terraform block stitches together a handful of resources:

aws_lambda_function wired to the zipped Python file with python3.9 runtime, 10s timeout, 128MB memory.
An IAM role with an sts:AssumeRole trust to lambda.amazonaws.com.
A CloudWatch log group at /aws/lambda/<name> with 14-day retention.
Two IAM policies attached to the role: one for CloudWatch Logs (CreateLogGroup, CreateLogStream, PutLogEvents) and one scoped to the S3 bucket ARN for object writes.
An aws_lambda_permission granting sns.amazonaws.com the right to invoke the function, scoped via source_arn to the specific SNS topic.

Gotcha: the log group needs a depends_on from the Lambda resource so it doesn't get created automatically (and then collide with the explicit one) on first apply.

Note

The S3 policy uses "s3:*" for simplicity. In production, follow the principle of least privilege and restrict to only "s3:PutObject" for the specific bucket.

Create SNS Topic and Subscription

SNS side is three resources: the topic, a Lambda subscription, and a topic policy with two statements. The first statement is the standard owner-only management statement (Subscribe, SetTopicAttributes, etc., conditioned on AWS:SourceOwner matching the account). The second statement allows events.amazonaws.com to call sns:Publish on the topic, which is the bit that lets EventBridge drop messages in.

Create EventBridge Rule

The EventBridge rule pattern filters on source: aws.ec2, detail-type: EC2 Instance State-change Notification, and a list of states (pending, stopping, stopped, shutting-down, terminated). The target points at the SNS topic ARN.

Quick gotcha: use terminated, not terminate, in the state list. The original cut of this had the wrong value and silently dropped terminate events.

Pro Tip

You can extend this pattern to capture other AWS service events like RDS snapshots, Auto Scaling activities, or EBS volume changes by modifying the EventBridge rule pattern.

Troubleshooting

Common Issues and Solutions

Lambda Not Receiving Events

Problem: EC2 state changes occur but no files appear in S3
Solution: Check that the SNS topic policy allows EventBridge to publish messages. Verify the Lambda permission allows SNS invocation.

S3 PutObject Access Denied

Problem: Lambda executes but fails with AccessDenied error
Solution: Verify the Lambda role has the S3 policy attached and the bucket name in the policy matches your actual bucket.

EventBridge Rule Not Triggering

Problem: The rule exists but events are not captured
Solution: Ensure the event pattern matches the exact format. Use "terminated" instead of "terminate" for the terminated state. Check CloudWatch Logs for the rule.

Lambda Timeout Errors

Problem: Lambda times out when writing to S3
Solution: Increase the timeout value (currently 10 seconds). Check if the Lambda is in a VPC and has proper NAT gateway access to S3.

Terraform Apply Failures

Problem: Circular dependency errors during terraform apply
Solution: Use depends_on to explicitly define resource dependencies. Ensure the Lambda zip file exists before applying.

Conclusion

In conclusion, using lifecycle events provides a powerful means of automation and orchestration in cloud computing. Through this blog post, we explored a specific use case: creating a text file in an S3 bucket triggered by an EC2 lifecycle event. By understanding the underlying concepts, setting up the necessary components, and following the step-by-step guide using AWS services, we have successfully demonstrated how to achieve this automation. The knowledge gained from this tutorial enables you to create event-driven automated triggers and take actions based on lifecycle events of other services as well as the ability to integrate with other applications. By embracing the principles of automation and eliminating repetition through tools like Terraform, you can enhance the efficiency and scalability of your applications in the cloud computing landscape.

Want Help With This?

If you're working on something similar and want a second set of eyes, or you'd like to talk through how this applies to your environment, reach out via the contact form. Happy to help.