Preventing unwanted changes and data loss in a CloudFormation stack

If you’re building serverless apps on AWS, it’s hard to avoid CloudFormation since most of the popular deployment frameworks (Serverless Framework, CDK, SAM, etc) are just a layer over the top of it.

CloudFormation is great for performing atomic updates to a set of cloud resources, but sometimes a stack can get into a bad state or cause unwanted changes and even data loss if the template is configured incorrectly.

Specifically, a CloudFormation stack update could:

1. Delete a resource which was accidentally removed by a developer from the CloudFormation template source
2. Delete a resource if a stack was accidentally deleted
3. Delete a resource whose attribute was changed in the CloudFormation template source. This would happen if the attribute that was updated was marked as “Update Requires: Replacement” in the CloudFormation documentation. The TableName field on a DynamoDB Table is an example of this.

To prevent these from happening, you can add the following 2 fields to your CloudFormation template alongside the resource you wish to protect:

1. DeletionPolicy — Set this to Retain to ensure that the resource will not be deleted if it’s removed from a stack or if its stack is deleted.
2. UpdateReplacePolicy — Set this to Retain to ensure that if a stack update requires replacing the resource, then the original instance of the resource will be retained even though a new one is created. If this happens and the new resource still uses a same unique field (e.g. name) that the old one had, then the new resource won’t be created and the stack update will fail.

My rules of thumb for setting these attributes:

• Apply them to all data-storing resources. This means DynamoDB Tables, S3 Buckets, Cognito User Pools, etc.
• Only use them in non-ephemeral environments, e.g. if you spin up a stack solely for a CI/CD test run and then delete it afterwards, you don’t want to leave any resources behind.

Here’s a code example of how to enable these settings in staging and prod stages using the Serverless Framework:

# serverless.yml
custom:
  deletionPolicy:
    default: Delete
    staging: Retain
    prod: Retain
  updateReplacePolicy:
    default: Delete
    staging: Retain
    prod: Retain

resources:
  Resources:
    MainTable:
      Type: AWS::DynamoDB::Table
      DeletionPolicy: ${self:custom.deletionPolicy.${self:provider.stage}, '${self:custom.deletionPolicy.default}'}
      UpdateReplacePolicy: ${self:custom.updateReplacePolicy.${self:provider.stage}, '${self:custom.updateReplacePolicy.default}'}

This tip was originally published here in my public notes vault.

—Paul