Testing Serverless Backends survey results + the local vs cloud debate

I got a great response to my Serverless Testing Survey I sent out on Friday (over 100 replies so far) and I wanted to share some of the main findings with you. If you completed the survey or shared on social media, I really appreciate it! 🙏

You’ve given me the reassurance I needed to proceed with writing the book—I started compiling a TOC and writing the first chapter yesterday 🙀.

On to the survey findings…

How satisfied are you with your approach to testing AWS serverless applications in your place of work?

(enable images to view chart)

Clearly a lot of room for improvement here.

What is your number 1 problem with testing serverless backends?

Answers here were the most interesting with “We don’t do any” and “where to start?” getting several mentions.

The most common theme was that of local testing. Folks were either struggling to do local testing well or were frustrated that others on their team insisted on doing local testing over cloudside testing.

This is a hot topic right now. This Twitter convo from yesterday provides interesting opposing views on the local vs cloudside debate and the original article by Gareth McCumskey on Why local development for serverless is an anti-pattern is definitely worth checking out.

My view is that fully local development/testing can be an XY Problem for some folks who use “local” as a proxy for both speed of feedback and isolation from other developers. These are undoubtedly two crucial attributes for a productive developer workflow and local has always been a reliable way to guarantee these.

But isolated cloud dev environments can be easily solved by individual developers deploying their own stacks (or AWS accounts) and there are techniques for bridging the feedback speed gap with cloudside testing, which I expect to get faster over time. The limitations of local-only integration testing—lack of fidelity between environments and restrictions to using only locally emulatable services—make it not worth the effort for backend development in my experience. (Markus Tacker has a good talk which goes deeper on this)

Most important AWS services for testing

Back to the survey, I asked “How important to you is being able to test your integrations with each of the following AWS services?”. The following are the mean responses (ranked from 1-Not Important At All to 6-Extremely Important):

• Lambda: 5.0
• DynamoDB: 4.5
• EventBridge/SNS: 4.3
• SQS: 4.2
• AWS API Gateway: 4.2
• S3: 4.1
• Step Functions: 3.8
• DynamoDB streams: 3.8
• Cognito: 3.6
• AppSync: 2.8

This question was to help me prioritise which service integrations to focus on writing chapters for. AppSync is one service I don’t cover in my testing workshop syllabus so would require the most research on my part. Given its lower popularity here and the existence of Yan Cui’s excellent AppSync Masterclass course, I may hold off on this for the first draft.

Before I go, I have one question for you—what is the ONE book in the software engineering/cloud space that had the greatest impact on you which you would most recommend to other developers? Hit reply and let me know.

Cheers,

—Paul