Handling future changes to your DynamoDB access patterns

My friend and fellow solo AWS consultant Adam Elmore posted this tweet yesterday which sparked several interesting replies:

“The worst part about DynamoDB is that you have to do some planning up front. The best part is literally everything else.”

The upfront planning to which Adam is referring is first understanding all the access patterns that your app will need to your database and then using these access patterns as the input to design the necessary table and GSI key schemas.

A common objection in the tweet replies was along the lines of “we simply won’t know all future user needs up front”. And this is indeed very true for most new software products. So the question then becomes “what do you do when your access patterns change?”.

Alex DeBrie has dedicated an entire chapter on migration strategies in his DynamoDB Book to answering this exact question. I’ll attempt a crude summary below.

When you need to accommodate a new access pattern, the actions you need to take in order to implement it will fall into one of the following brackets (in increasing order of effort):

1. Update your data access code only. Example use cases: adding new attributes to an existing entity, adding a new entity type without relations or adding a new entity type under an existing parent item collection.
2. Add a new GSI. Whenever the new access pattern can be achieved by creating a new GSI that references pre-existing item attributes in its key schema.
3. Change data within existing items to support new GSI. Whenever a GSI requires creation of new composite attributes as its keys (say gsi1pk and gsi1sk) AND existing data items need to be present within this index. This requires running a scan script across your DynamoDB table to set these new fields on all existing items.

The requirement to filter or sort existing items in a new way could be implemented by 1, 2 or 3, and will depend on the data attributes and GSIs that are already present and whether or not client-side parsing could suffice.

So the worst case (in terms of engineering effort) is that you need to run a scan script to update data in existing items. In my experience helping many startups build serverless apps on top of DynamoDB, this has been extremely rare. The vast majority of changes can be accomplished by 1 or 2.

I strongly recommend you check out Chapter 15 of the DynamoDB Book for detailed worked examples of all the different change scenarios that could arise and how you can deal with them.