How to prevent concurrent deployments of serverless stacks in GitHub Actions

Having previously used AWS CodePipeline+CodeBuild for the Continuous Deployment pipelines for my serverless apps, I switched to GitHub Actions last year, mainly because it’s much easier to setup and is more approachable for application developers on a small team who are already using GitHub.

One of the features that AWS CodePipeline provides out-of-the-box is the automatic queueing of new commits while an existing stage is still in progress. This does not happen automatically in GitHub Actions and requires a little extra code to be added to your workflow file.

Let’s illustrate this problem with a CD pipeline that is triggered on pushes to the main branch. It runs some local unit tests and then proceeds to deploy the application stack(s) to test, staging and prod environments in series, running post-deployment integration/E2E tests at each stage.

In this pipeline, consider the scenario where Developer A merges a PR into main triggering a new workflow execution. A few seconds later, Developer B does the same with a separate PR. Given that deployment steps can take a minute or two to complete, there is a high likelihood that if run concurrently, the execution for Developer B would fail with a CloudFormation error stating that a stack update is already in progress. Or equally bad, the deployment may not happen concurrently but the second execution may start its deployment while the integration/E2E test run for Developer A’s commit is still in progress.

So we need to ensure that each workflow execution has exclusive access to a particular stage for deployment and integration/E2E testing. Any incoming commits should be queued until a stage completes (either successfully or with a failure).

When implementing this pipeline as a GitHub Actions workflow, each stage can be defined as a sequential Job.

The following abbreviated GHA workflow file shows the key configuration settings you need to set this up:

# .github/workflows/main-api.yml
name: main-api

on:
  push:
    branches:
      - main
    paths:
      - 'services/api/**'

jobs:
  build:
    steps:  # omitted for brevity
  
  deploy-test:
    needs: build
    concurrency:
      group: ${{ format('{0}-{1}', github.workflow, github.job) }}
    steps:  # omitted for brevity

  deploy-staging:
    needs: deploy-test
    concurrency:
      group: ${{ format('{0}-{1}', github.workflow, github.job) }}
    steps:  # omitted for brevity

  deploy-prod:
    needs: deploy-staging
    concurrency:
      group: ${{ format('{0}-{1}', github.workflow, github.job) }}
    steps:  # omitted for brevity
    

The key things to note here are:

• the needs attribute: by default, jobs are run in parallel so this attribute ensures they are run in a defined sequence
• the concurrency.group attribute: this acts as a hash key which GitHub Actions uses to track in-progress jobs. Here, I’m using the combination of the workflow name (main-api) and the job name (deploy-test, etc). (Read more here).

In my earlier scenario of Developer A pushing to main just before Developer B, once the workflow executions for both commits are triggered, the build job for both will still proceed concurrently (since this doesn’t touch an AWS environment, we have no concurrency concerns here). However, once Developer A’s execution hits the deploy-test job, GHA will internally mark it as in-progress and once Developer B’s execution reaches the same job, it will have its status set to Queued until A’s execution proceeds to the deploy-staging job.

This approach also works well in a monorepo where independently deployable services in separate subfolders have their own workflow pipelines. By including the workflow name as part of the concurrency group, each workflow can maintain its own isolated concurrency group for each target environment.