Preventing unwanted changes and data loss in a CloudFormation stack

If you’re building serverless apps on AWS, it’s hard to avoid CloudFormation since most of the popular deployment frameworks (Serverless Framework, CDK, SAM, etc) are just a layer over the top of it.

CloudFormation is great for performing atomic updates to a set of cloud resources, but sometimes a stack can get into a bad state or cause unwanted changes and even data loss if the template is configured incorrectly.

Specifically, a CloudFormation stack update could:

  1. Delete a resource which was accidentally removed by a developer from the CloudFormation template source
  2. Delete a resource if a stack was accidentally deleted
  3. Delete a resource whose attribute was changed in the CloudFormation template source. This would happen if the attribute that was updated was marked as “Update Requires: Replacement” in the CloudFormation documentation. The TableName field on a DynamoDB Table is an example of this.

To prevent these from happening, you can add the following 2 fields to your CloudFormation template alongside the resource you wish to protect:

  1. DeletionPolicy — Set this to Retain to ensure that the resource will not be deleted if it’s removed from a stack or if its stack is deleted.
  2. UpdateReplacePolicy — Set this to Retain to ensure that if a stack update requires replacing the resource, then the original instance of the resource will be retained even though a new one is created. If this happens and the new resource still uses a same unique field (e.g. name) that the old one had, then the new resource won’t be created and the stack update will fail.

My rules of thumb for setting these attributes:

  • Apply them to all data-storing resources. This means DynamoDB Tables, S3 Buckets, Cognito User Pools, etc.
  • Only use them in non-ephemeral environments, e.g. if you spin up a stack solely for a CI/CD test run and then delete it afterwards, you don’t want to leave any resources behind.

Here’s a code example of how to enable these settings in staging and prod stages using the Serverless Framework:

# serverless.yml
    default: Delete
    staging: Retain
    prod: Retain
    default: Delete
    staging: Retain
    prod: Retain

      Type: AWS::DynamoDB::Table
      DeletionPolicy: ${self:custom.deletionPolicy.${self:provider.stage}, '${self:custom.deletionPolicy.default}'}
      UpdateReplacePolicy: ${self:custom.updateReplacePolicy.${self:provider.stage}, '${self:custom.updateReplacePolicy.default}'}

This tip was originally published here in my public notes vault.


Join daily email list

I publish short emails like this on building software with serverless on a daily-ish basis. They’re casual, easy to digest, and sometimes thought-provoking. If daily is too much, you can also join my less frequent newsletter to get updates on new longer-form articles.

    View Emails Archive

    Architecture & Process Review

    Built a serverless app on AWS, but struggling with performance, maintainability, scalability or DevOps practices?

    I can help by reviewing your codebase, architecture and delivery processes to identify risk areas and their causes. I will then recommend solutions and help you with their implementation.

    Learn more >>

    🪲 Testing Audit

    Are bugs in production slowing you down and killing confidence in your product?

    Get a tailored plan of action for overhauling your AWS serverless app’s tests and empower your team to ship faster with confidence.

    Learn more >>