When prototype becomes production

The biggest problem in software engineering is that it’s easy to make something that kinda-sorta works OK, and incredibly difficult to make something that works 100% perfectly. Our industry rewards people for coming up with ideas and prototypes, and punishes them for the actual hard work of delivering quality – Daniel Feldman on Twitter

While I don’t think “works 100% perfectly” should be an aspiration for the vast majority of software teams, I agree with the general sentiment here.

The prototyping process necessarily favours speed and low-cost over quality, scalability and completeness. You need to get a feature in front of users for feedback to see if an idea is even viable. Often what’s being built is not even considered to be a product yet.

The nice thing with serverless architectures is that these compromises aren’t as significant as they would’ve been in the past. You can quickly provision autoscalable services and only pay for how much or little you use them. So the building blocks that you use for the prototype app are often the exact same as those for the production app.

But this ease of setup, low-cost and built-in scalability can lead to the temptation of treating the prototype system as the production system, especially if the prototype was built by developers with little operational experience.

There are a few “high-gravity” prototyping decisions that you can make early on which would involve significant rework to fix that are worth being aware of:

• Co-locating different applications in the same AWS account. Once a production system is in live usage in an AWS account, it’s very difficult to move it. There are several reasons why using a separate AWS account is a good idea, including “noisy neighbour” isolation, security, cost accounting and if you ever wish to sell an individual application.
• Choice of database. Similar to the previous point, as soon as you start storing real production user data, it now gets much more difficult to change where and how it’s being stored. So if you choose an RDS database to prototype with purely because of familiarity with SQL, and you then decide that the operational aspects of DynamoDB better fit your needs, this is going to be a time-consuming and risky migration.
• Manually provisioning cloud resources via the console. If you didn’t use Infrastructure-as-Code, automating releases via CI/CD pipelines becomes much more difficult. While you could do this retrospectively since CloudFormation now allows you to bring existing resources into your stacks, this can get messy with lots of conditional environment-specific logic that is hard to reason about.

Have you had any prototype applications turn into production systems? If so, what would you do differently if you had your chance to start it again?