How to prevent concurrent deployments of serverless stacks in GitHub Actions

Having previously used AWS CodePipeline+CodeBuild for the Continuous Deployment pipelines for my serverless apps, I switched to GitHub Actions last year, mainly because it’s much easier to setup and is more approachable for application developers on a small team who are already using GitHub.

One of the features that AWS CodePipeline provides out-of-the-box is the automatic queueing of new commits while an existing stage is still in progress. This does not happen automatically in GitHub Actions and requires a little extra code to be added to your workflow file.

Let’s illustrate this problem with a CD pipeline that is triggered on pushes to the main branch. It runs some local unit tests and then proceeds to deploy the application stack(s) to test, staging and prod environments in series, running post-deployment integration/E2E tests at each stage.

mainline CD pipeline

In this pipeline, consider the scenario where Developer A merges a PR into main triggering a new workflow execution. A few seconds later, Developer B does the same with a separate PR. Given that deployment steps can take a minute or two to complete, there is a high likelihood that if run concurrently, the execution for Developer B would fail with a CloudFormation error stating that a stack update is already in progress. Or equally bad, the deployment may not happen concurrently but the second execution may start its deployment while the integration/E2E test run for Developer A’s commit is still in progress.

So we need to ensure that each workflow execution has exclusive access to a particular stage for deployment and integration/E2E testing. Any incoming commits should be queued until a stage completes (either successfully or with a failure).

When implementing this pipeline as a GitHub Actions workflow, each stage can be defined as a sequential Job.

The following abbreviated GHA workflow file shows the key configuration settings you need to set this up:

# .github/workflows/main-api.yml
name: main-api

on:
  push:
    branches:
      - main
    paths:
      - 'services/api/**'

jobs:
  build:
    steps:  # omitted for brevity
  
  deploy-test:
    needs: build
    concurrency:
      group: ${{ format('{0}-{1}', github.workflow, github.job) }}
    steps:  # omitted for brevity

  deploy-staging:
    needs: deploy-test
    concurrency:
      group: ${{ format('{0}-{1}', github.workflow, github.job) }}
    steps:  # omitted for brevity

  deploy-prod:
    needs: deploy-staging
    concurrency:
      group: ${{ format('{0}-{1}', github.workflow, github.job) }}
    steps:  # omitted for brevity
    

The key things to note here are:

  • the needs attribute: by default, jobs are run in parallel so this attribute ensures they are run in a defined sequence
  • the concurrency.group attribute: this acts as a hash key which GitHub Actions uses to track in-progress jobs. Here, I’m using the combination of the workflow name (main-api) and the job name (deploy-test, etc). (Read more here).

In my earlier scenario of Developer A pushing to main just before Developer B, once the workflow executions for both commits are triggered, the build job for both will still proceed concurrently (since this doesn’t touch an AWS environment, we have no concurrency concerns here). However, once Developer A’s execution hits the deploy-test job, GHA will internally mark it as in-progress and once Developer B’s execution reaches the same job, it will have its status set to Queued until A’s execution proceeds to the deploy-staging job.

This approach also works well in a monorepo where independently deployable services in separate subfolders have their own workflow pipelines. By including the workflow name as part of the concurrency group, each workflow can maintain its own isolated concurrency group for each target environment.

Join daily email list

I publish short emails like this on building software with serverless on a daily-ish basis. They’re casual, easy to digest, and sometimes thought-provoking. If daily is too much, you can also join my less frequent newsletter to get updates on new longer-form articles.

    View Emails Archive

    Free Intro Call

    Book a free 30-minute introduction call with me to see how we could work together.

    Select a time for our call

    🪲 Testing Audit

    Are bugs in production slowing you down and killing confidence in your product?

    Get a tailored plan of action for overhauling your AWS serverless app’s tests and empower your team to ship faster with confidence.

    Learn more >>