Evolving DynamoDB Designs

Conventional wisdom seems to be that changing a DynamoDB design is extremely difficult and you want to avoid it at all costs.

It is repeated without question that you must determine all your access patterns up front because you cannot easily change your DynamoDB design after the fact. For example:

“you cannot change primary indexes so the only option is to create a new (table), migrate the data with transformations and delete the old table. This process is excruciating, especially in production.”

However, with DynamoDB single table designs, this thinking is not the whole picture, and can be misleading.

When using single-table DynamoDB designs, your keys and attributes are uncoupled from the physical implementation, so you can evolve your DynamoDB design much more easily.

This may be the most important benefit of single-table designs.

While planning your query access patterns is hugely important, even essential, this does not mean that the only way to change a DynamoDB implementation is a “fork-lift” upgrade where you are forced create a new table.

Over the past year, we have extensively evolved our SenseDeep DynamoDB implementation without any downtime. We have not once had to re-create our DynamoDB table or modify our primary keys. We have added new entities, new access patterns, new indexes, changed inter-entity references and extended our item collections.

We did this by using single-table designs that uncoupled our keys and attributes from the physical table keys and attributes. This is an essential tenet of single-table designs. We used the DynamoDB OneTable Library that makes defining your entities and attributes and uncoupling your keys straight forward.

For secondary indexes, we used attribute packing to project attributes from multiple entities into a single GSI data attribute. This further uncouples index design from the actual data schema.

We use sequenced, reversible DynamoDB migrations to evolve our DynamoDB design without downtime. These migrations are small, scriptable atomic database mutations that add new capabilities before removing or modifying existing attributes or entities. Using migrations expressed as code, you can confidently evolve your DynamoDB design knowing that you can easily and quickly reverse a change.

We use the SenseDeep Migration Manager and the OneTable CLI to sequence and manage (or rollback) database migrations interspersed with code deploys.

SenseDeep with OneTable

At SenseDeep, we’ve used OneTable and the OneTable CLI extensively with our SenseDeep serverless developer studio. All data is stored in a single DynamoDB table and we extensively use single-table design patterns. We could not be more satisfied with DynamoDB implementation. Our storage and database access costs are insanely low and access/response times are excellent.

More?

You can read more about the Migration Manager at SenseDeep MigrationManager, OneTable at: OneTable and OneTable CLI.

Links

• SenseDeep DynamoDB Studio
• OneTable Post
• OneTable Migrate CLI
• SenseDeep App