CloudTrail-CloudWatch-S3 Integration for Enhanced Monitoring

A Guide with AWS CloudShell

Creating a CloudTrail in a hurry can be challenging when using the AWS Management Console. But don't worry, there's a quicker way. In this guide, I'll show you how to quickly create a CloudTrail and send its log data to an Amazon S3 bucket and a CloudWatch Log Group using a script in the AWS CloudShell environment.

To level set, here are some services that we will touch:

• CloudShell
• CloudTrail
• S3
• CloudWatch
• Roles
• Policies

AWS CloudShell is a browser-based shell that comes pre-authenticated. You can run AWS CLI commands directly from the AWS Management Console using your preferred shell, such as Bash, PowerShell, or Z shell, without the need to download or install any tools.

AWS CloudTrail is a powerful service that enables operational auditing, governance, and compliance in your AWS account. It records all actions taken by users, roles, or AWS services in the form of events. These events can include actions taken in the AWS Management Console, AWS CLI, and AWS SDKs and APIs.

Amazon S3 is a scalable, secure, and highly-available object storage service. It's used by organizations of all sizes for a variety of use cases, including data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. With S3, you have the ability to manage and control access to your data to meet your specific business, organizational, and compliance requirements.

Amazon CloudWatch is a real-time monitoring service for AWS resources and applications. It collects and tracks metrics to give you a complete view of your resources and applications. CloudWatch Logs consist of log groups and log streams. A log stream is a sequence of log events that share the same source, while a log group is a collection of log streams that share the same retention, monitoring, and access control settings. There is no limit on the number of log streams that can belong to a log group.

Code (with comments)

#create a log group called 'arun-logs'
aws logs create-log-group --log-group-name 'arun-logs'

#create a s3 bucket called arun0213b
aws s3 mb s3://arun0213b

#create a bucket policy file called bucketpolicy.json which allows the CloudTrail service to write to s3 bucket
echo '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AWSCloudTrailWrite",
            "Effect": "Allow",
            "Principal": {
              "Service": "cloudtrail.amazonaws.com"
            },
            "Action": "s3:*",
            "Resource": ["arn:aws:s3:::arun0213b", "arn:aws:s3:::arun0213b/*"]
        }
    ]
}' > bucketpolicy.json

#applies the policy to the s3 bucket
aws s3api put-bucket-policy --bucket arun0213b --policy file://bucketpolicy.json
#create a a role policy file called trailrole.json which allows CloudTrail to assume a role
echo '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudtrail.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}' > trailrole.json

#create an IAM role called trailrole and applies the role policy file
aws iam create-role --role-name trailrole --assume-role-policy-document file://trailrole.json

#create a policy document which will be applied to a customer managed policy
echo '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}' > trailpolicy.json

#create a customer managed policy and applies the policy document file
aws iam create-policy --policy-name trailpolicy --policy-document file://trailpolicy.json

#create a variable for the customer managed policy arn
trailpolicy_arn=$(aws iam list-policies --query 'Policies[?PolicyName==`trailpolicy`].Arn' --output text)

#attach the customer managed policy (via the arn variable) to the IAM role created earlier
aws iam attach-role-policy --role-name trailrole --policy-arn $trailpolicy_arn

#create a variable for trail role arn
trailrole_arn=$(aws iam get-role --role-name trailrole --query 'Role.Arn' --output text)

#create a variable for the log group arn
loggroup_arn=$(aws logs describe-log-groups --log-group-name-prefix arun-logs --query 'logGroups[0].arn' --output text)

#create a CloudTrail and write the contents to the s3 bucket and log group using the trail role
aws cloudtrail create-trail --name arun0213cloudtrail --s3-bucket-name arun0213b --no-is-multi-region-trail --cloud-watch-logs-role-arn $trailrole_arn --cloud-watch-logs-log-group-arn $loggroup_arn

#start CloudTrail log
aws cloudtrail start-logging --name arun0213cloudtrail

aws logs create-log-group --log-group-name 'arun-logs'
aws s3 mb s3://arun0213b
echo '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AWSCloudTrailWrite",
            "Effect": "Allow",
            "Principal": {
              "Service": "cloudtrail.amazonaws.com"
            },
            "Action": "s3:*",
            "Resource": ["arn:aws:s3:::arun0213b", "arn:aws:s3:::arun0213b/*"]
        }
    ]
}' > bucketpolicy.json
aws s3api put-bucket-policy --bucket arun0213b --policy file://bucketpolicy.json
echo '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudtrail.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}' > trailrole.json
aws iam create-role --role-name trailrole --assume-role-policy-document file://trailrole.json
echo '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}' > trailpolicy.json
aws iam create-policy --policy-name trailpolicy --policy-document file://trailpolicy.json
trailpolicy_arn=$(aws iam list-policies --query 'Policies[?PolicyName==`trailpolicy`].Arn' --output text)
aws iam attach-role-policy --role-name trailrole --policy-arn $trailpolicy_arn
trailrole_arn=$(aws iam get-role --role-name trailrole --query 'Role.Arn' --output text)
loggroup_arn=$(aws logs describe-log-groups --log-group-name-prefix arun-logs --query 'logGroups[0].arn' --output text)
aws cloudtrail create-trail --name arun0213cloudtrail --s3-bucket-name arun0213b --no-is-multi-region-trail --cloud-watch-logs-role-arn $trailrole_arn --cloud-watch-logs-log-group-arn $loggroup_arn
aws cloudtrail start-logging --name arun0213cloudtrail

Troubleshooting