7 ways to do async message processing in AWS

Last time we talked about why you should aim to use asynchronous message processing liberally within your application and how it’s now much easier to do so using serverless technologies.

In this article, I want to show you how you can do this in AWS. I will give you a quick run-through of the main services you can use to implement async messaging and when you might want to choose each one for your particular workflow.

Simple Queue Service (SQS)

SQS is a managed message queue service. It’s fully integrated into the Lambda service so rather than needing to write code to poll the queue, you simply configure your Lambda function to be triggered by new messages in the queue and AWS will invoke it for you.

SQS could be a good choice for your async workflow if any of the following apply:

• You only need to perform a single processing task for each message within the same microservice and don’t need PubSub capability.
• You want the ability to process multiple messages within a single Lambda function invocation.
• You are expecting high throughput and your processing function talks to downstream services that may be susceptible to overloading (e.g. a SQL database or a rate-limited third party API). You can configure a concurrency level on the processing Lambda function in order to effectively throttle calls to the downstream services.
• You want processing failures to be automatically retried.
• You need to ensure messages are processed in the order they are received (use a First-in-first-out (FIFO) queue).

Directly invoked Lambda functions with InvocationType=Event

A lesser-known feature of Lambda is the ability to invoke a function directly from another function using the Lambda SDK and passing InvocationType=Event as a parameter along with an event payload. The Lambda service simply sends an acknowledgement response to the calling function and then asynchronously triggers an execution of the target processing function.

This approach could work well for you if:

• You don’t need PubSub capability.
• You want processing failures to be automatically retried.
• You don’t want the overhead of configuring another cloud resource (e.g. an SNS topic or SQS queue).
• Your processing function only needs to receive one message at a time (as opposed to a batch).

Simple Notification Service (SNS)

SNS is a Publish-Subscribe (PubSub) messaging service. Like SQS, it’s fully integrated into the Lambda service so rather than needing to write code to poll the topic, you simply configure your Lambda function to be triggered by new messages sent to the topic and AWS will invoke it for you.

Using SNS could be a good approach for you if:

• Your processing function only needs to receive one message at a time (as opposed to a batch).
• You want to pass a message between loosely coupled microservices.
• There are potentially several independent tasks you need to perform in response to a single application event (e.g. new user created).
• You want processing failures to be automatically retried.
• Your processing function only needs to process a subset of message types (using configurable message filters).

EventBridge

“Amazon EventBridge is a serverless event bus that makes it easy to connect applications together using data from your own applications”

EventBridge is similar to SNS in that it also acts like a PubSub service. Consider using EventBridge if:

• Your processing function only needs to receive one message at a time (as opposed to a batch).
• You want to pass a message between loosely coupled microservices.
• There are potentially several independent tasks you need to perform in response to a single application event (e.g. new user created).
• You want processing failures to be automatically retried.
• Your processing function only needs to process a subset of message types (using configurable message filters).
• You need fine-grained control over how to filter messages before passing to a processing function (by using message body fields rather than just envelope-level message attributes).
• You want to consume events from supported third party providers.
• You want to publish events to other external systems (e.g. if you’re building a SaaS product).
• You want greater monitoring visibility into your message flow.

DynamoDB streams

DynamoDB streams enable changes in data within a DynamoDB table to be automatically passed to a Lambda function in the form of a message batch.

Consider using DynamoDB streams if:

• Your existing workflow already requires writing an item to DynamoDB.
• You want the ability to process multiple messages within a single Lambda function invocation.
• You don’t mind that your processing function will need an understanding of the DynamoDB item schema in order to parse it (as opposed to a more friendly domain event schema).
• You are ok with having all changes passed unfiltered into your Lambda function and having your function code itself do any required filtering.
• You want multiple separate functions to be able to independently process changes to the database.
• You want processing failures to be automatically retried.

Kinesis Data Streams

Kinesis is a real-time data streaming service. Consider using it if:

• Your workload involves ingesting a high volume of events from external sources (e.g. web clickstreams or IoT).
• You want the ability to process multiple messages within a single Lambda function invocation.
• You are ok with having all changes passed unfiltered into your Lambda function and having your function code itself do any required filtering.
• You want multiple separate functions to be able to independently process messages.
• You want processing failures to be automatically retried.
• You don’t mind having to do a bit of upfront capacity planning to configure what shard size you’ll need.
• You don’t mind paying by the hour rather than per-use.

Lambda Destinations

Lambda Destinations allow you to chain “publishing” and “consuming” Lambda functions together asynchronously.

Consider using Lambda Destinations if:

• Your publishing function has already been invoked asynchronously (e.g. in response to an SNS event).
• You want to initiate a more complex workflow upon completion of your consuming function’s execution (e.g. by publishing a new SNS event).
• You want to manage orchestration of your calling chain in config code (YAML) rather than imperative Lambda function code.
• You are ok with always having your consuming Lambda function trigger for every completed execution of the publishing function.
• You want processing failures to be automatically retried.

Composing services together for more complex flows

So far I’ve covered how to use a single service (along with Lambda) to achieve a simple asynchronous workflow. It is possible to create more complex and powerful flows by composing multiple services together.

To learn more on this, I highly recommend you read Jeremy Daly’s article on Serverless Microservice Patterns for AWS. You may also wish to considering using AWS Step Functions to co-ordinate a more complex flow.

— Paul