From Source to Destination: The Art of DynamoDB Migration

Introduction

AWS DynamoDB is a managed NoSQL database service provided by Amazon Web Services, designed to offer seamless scalability, high performance, and availability. Unlike traditional relational databases, which use structured schema with tables, rows, and columns, NoSQL databases like DynamoDB are schema-less. This makes them exceptionally flexible and adaptable to varying data requirements, especially for applications that need to scale dynamically.

The popularity for this service varies; however the main advantages revolve around:

• its ability to scale automatically without any downtime or performance degradation
• it is a fully managed service by AWS (no more admistration work)
• replicates data across multiple AWS data centers, ensuring high availability
• easy integrated with other AWS services
• less costly (compared to other license modeled databases), as you only pay for the capacity you use, and there are no upfront fees

One more item to note in terms of popularity is how easy the process will be to clone or even migrate a DynamoDB table(s) somewhere else, including a whole new account. This post explains the steps needed to accomplish the migration to another AWS account.

Prerequisites

• Access to the source AWS account and the destination AWS account
• Permissions to:
  • create a S3 bucket in the target AWS account
  • backup the source DynamoDB table
  • create and restore the DynamoDB table in the target AWS Account
  • access the AWS CLI or CloudShell

Logical

Steps

Destination Account – Create S3 Bucket and Add Bucket Policy

Create S3 Bucket For Backup

#sets the variable for the bucket name
BUCKET_NAME="dynamodb-backup"
#creates the bucket and gives the name you set in the variable above
aws s3api create-bucket --bucket $BUCKET_NAME --region us-east-1 

#creates a bucketpolicy.json file that allows the source & destination accounts to access the bucket
echo "{
    \"Version\": \"2012-10-17\",
    \"Statement\": [
        {
            \"Effect\": \"Allow\",
            \"Principal\": {
              \"AWS\": [
                \"sourceAWSAccountID\",
                \"destinationAWSAccountID\"
                 ]
            },
            \"Action\": \"s3:*\",
            \"Resource\": [\"arn:aws:s3:::$BUCKET_NAME\", \"arn:aws:s3:::$BUCKET_NAME/*\"]
        }
    ]
}" > bucketpolicy.json

#puts the bucket policy defined in teh json file to the bucket
aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://bucketpolicy.json

Source Account – Backup DB

Backup DynamoDB

#backups up the DynamoDB called tblARUN to the s3 bucket created above in the destination account

aws dynamodb export-table-to-point-in-time --region us-east-1 --table-arn arn:aws:dynamodb:us-east-1:[sourceAWSAccountID]:table/tblARUN --s3-bucket dynamodb-backup --s3-bucket-owner [destinationAWSAccountID]

This will result in a json export of the task, which will contain the ARN of the task. You can use that to check the status:

aws dynamodb describe-export --export-arn [ARN output from backup command]

The backup job will create subfolders (called prefixes in S3 world) in the destination S3 bucket AWSDynamoDB/[random number]/data. You will have to find the actual path as you will need this for restore.

Once you see the ‘Completed’ status, you can move onto the restore task at the destination account.

Destination Account – Restore DB

Create Table Creation Parameters File

Create the json parameters file needed for table creation and called it dbparameters.json

echo '{
"TableName": "tblDANIEL", #this is the target table in the destination that will be created
"AttributeDefinitions": [
{
"AttributeName": "ID", #this will be the partition key
"AttributeType": "S" #the partion key will be a string
},
{
"AttributeName": "State", #this will be the sort key
"AttributeType": "S" #the sort key will be a string
}
],
"KeySchema": [
{
"AttributeName": "ID",
"KeyType": "HASH" #partition key
},
{
"AttributeName": "State",
"KeyType": "RANGE" #sort key
}
],
"BillingMode": "PROVISIONED",
"ProvisionedThroughput": {
"ReadCapacityUnits": 5,
"WriteCapacityUnits": 5
}
}' > dbparameters.json

Create Destination Table via S3 Import

As stated before, when a backup to S3 is completed, a specific path is created in the bucket. That path (or prefix) will have the format of AWSDynamoDB/[random numbers]/data. Make sure you know this path as you will need it for the below command.

aws dynamodb import-table --s3-bucket-source S3Bucket=dynamodb-backup,S3KeyPrefix=AWSDynamoDB/[random GUID]/data --input-format DYNAMODB_JSON --table-creation-parameters file://dbparameters.json --input-compression-type GZIP

Check Status

To check the status of the table restore, you can run the command:

aws dynamodb describe-table --tblDANIEL

Conclusion

In conclusion, AWS DynamoDB presents itself as a formidable choice for businesses looking for a flexible, scalable, and efficient NoSQL database solution. Whether you’re launching a startup or scaling an enterprise-level application, DynamoDB offers the features and performance to meet your data management needs. In addition to the numerous advantanges for this option, another big one is the capability to easily backup and restore your tables either within the same AWS account or separate AWS account.