The pains of testing serverless applications

Over one third (36.4%) of respondents to my serverless adoption survey cited Testing as one of their top 3 areas that caused them most pain or concern when building with serverless. It’s the second highest item, with Debugging/Monitoring out in front on 57.9% and Developer Workflow in third on 30.6%.

When you couple this statistic with the 39% of the same respondents who also rated “Faster Feature Development” as a “Very Important Factor” in their initial decision to choose a serverless architecture, you might begin to worry about a load of fastly developed, poorly tested serverless apps out there in the wild.

I’ve curated several of the raw freeform survey responses that mention testing below to give you a flavour of the specific problems folks are having:

• • •

“Local development and testing is still not mature enough which takes a lot of time to setup / maintain”

“Being a quality fanatic, testing cloud-native solutions really is a challenge.”

“Doing dev testing and debugging weren’t easy. We ended up doing it entirely in the cloud.”

“Testing, hard to test, as not each and every case can be done by object testing via lambda console, especially connections between the services, as overall microservices/serverless approach.”

“Developer testing still a pain.”

“Currently no testing can be done on local computers. It is uploaded to the cloud on test instances and tested and debugged there. Takes time from development and costs money.”

“Develop solution as micro services is complex to test, debug and deploy”

“I noticed that serverless projects which mainly used lambda functions and API Gateway had much lower quality controls, almost non-existent unit tests…”

“We also faced challenges testing the application locally as all the third-party plugins for local development like SAM local, sls-offline, etc didn’t meet our needs properly.”

“Some time testing took long time, because permission issues.”

“We don’t do local backend testing - each dev gets their own sandbox AWS account that they can safely deploy their changes to before merging to master. That’s the most ergonomic workflow I’ve found for testing serverless in order to avoid heavily mocking service integrations. It does make testing slow since CodePipeline, CodeBuild, CloudFormation are not the fastest at deployment.”

“Local testing is a fairly large challenge. Mainly to do with speed to test. And since complete Serverless applications are usually using mainly services with interconnecting functions, queues and dbs it’s nearly impossible to test locally in any meaningful way. You’re forced to test per function. Which has its benefits but is a fairly large learning curve.”

“It’s hard to be so much dependent on mocking cloud services for local development.”

“Serverless is far more difficult for traditional developers than anticipated. Traditional thinking of everything just works does not apply with serverless and developer mindset should set to handle uncertainty and retry.”

• • •

I have a few observations on these responses:

• Most refer to testing within an individual developer’s environment as opposed to tests running against other environments in a CI/CD pipeline, i.e. tests that developers are writing and running as part of their iterative feature dev workflow (before pushing to main branch).
• Speed (or lack thereof) of the testing feedback loop is explicit or implicit in many responses.
• Several folks have the desire/expectation (rightly or wrongly) to be able to run tests entirely locally on their workstation.
• “Local” seems to be being used as an analog to “fast”.
• Complexity inherent within distributed/microservice based systems slows down test authoring and increases test fragility—“how do I write an end-to-end test for this async data pipeline?”

Is testing serverless applications a major pain point for you or your team? Or do you have a testing strategy that is working particularly well?

If you answered yes to either of these I’d love it if you could hit reply and tell me a bit more about it. I’m at the early stages of planning a workshop for later this year and I expect testing to be large part of it.

— Paul