DynamoDB OneTable

plain-ring

DynamoDB OneTable (OneTable) is an access library for DynamoDB applications that use one-table design patterns with NodeJS. OneTable makes dealing with DynamoDB and one-table design patterns dramatically easier while still providing easy access to the full DynamoDB API.

OneTable is used by the SenseDeep Serverless Developer Studio for all DynamoDB access. OneTable is provided open source (MIT license) from GitHub OneTable or NPM OneTable.

History and Credits

rick-houlihan

After watching the famous Rick Houlihan DynamoDB ReInvent Video, we changed how we used DynamoDB for our SenseDeep serverless developer studio to use one-table design patterns. However, we found the going tough and thus this library was created to make our one-table patterns less tedious, more natural and a joy with DynamoDB.

A big thank you to Alex DeBrie and his excellent DynamoDB Book. Highly recommended.

OneTable Overview

plain-ring

OneTable is not an ORM. Rather it provides a convenience API over the DynamoDB APIs. It offers a flexible high-level API that supports one-table design patterns and eases the tedium of working with the standard, unadorned DynamoDB API.

OneTable can invoke DynamoDB APIs or it can be used as a generator to create DynamoDB API parameters that you can save or execute yourself.

OneTable is not opinionated (as much as possible) and provides hooks for you to customize requests and responses to suit your exact needs.

Here are some of the key features of OneTable

  • Schema supported one-table access to DynamoDB APIs.
  • Efficient storage and access of multiple entities in a single DynamoDB table.
  • High level API with type marshaling, validations, and extended query capability for get/delete/update operations.
  • Bidirectional conversion of DynamoDB types to Javascript types.
  • Option to invoke DynamoDB or simply generate API parameters.
  • Generation of Conditional, Filter, Key and Update expressions.
  • Schema item definitions for attribute types, default values, enums and validations.
  • Powerful field level validations with required and transactional unique attributes.
  • Easy parameterization of filter and conditional queries.
  • Multi-page response aggregation.
  • Compound and templated key management.
  • Encrypted fields.
  • Support for Batch, Transactions, GSI, LSI indexes.
  • Hooks to modify DynamoDB requests and responses and for item/attribute migrations.
  • Controllable logging to see exact parameter, data and responses.
  • Simple, easy to read source to modify (< 1000 lines).
  • Safety options to prevent "rm -fr *".
  • No module dependencies.
  • Support for the AWS SDK v3 and v2
  • TypeScript type inference from schema for full type validation on APIs, parameters, returns, and entities and attributes.

Installation

npm i dynamodb-onetable

Quick Tour

Import the OneTable library. If you are not using ES modules or Typescript, use require to import the libraries.

import {Table} from 'dynamodb-onetable'

If you are using the AWS SDK V2, import the AWS DynamoDB class and create a DocumentClient instance.

import DynamoDB from 'aws-sdk/clients/dynamodb'
const client = new DynamoDB.DocumentClient(params)

This version includes prototype support for the AWS SDK v3.

If you are using the AWS SDK v3, import the AWS v3 DynamoDBClient class and the OneTable Dynamo helper. Then create a DynamoDBClient instance and Dynamo wrapper instance.

import {DynamoDBClient} from '@aws-sdk/client-dynamodb'
import Dynamo from 'dynamodb-onetable/Dynamo'
const client = new Dynamo({client: new DynamoDBClient(params)})

Initialize your your OneTable Table instance and define your models via a schema. The schema defines your single-table entities, attributes and indexes.

const table = new Table({
    client: client,
    name: 'MyTable',
    schema: MySchema,
})

This will initialize your your OneTable Table instance and define your models via a schema.

Schemas

Schemas define your models (entities), keys, indexes and attributes. Schemas look like this:

const MySchema = {
    indexes: {
        primary: { hash: 'pk', sort: 'sk' }
        gs1:     { hash: 'gs1pk', sort: 'gs1sk' }
    },
    models: {
        Account: {
            pk:          { value: 'account:${name}' },
            sk:          { value: 'account:' },
            id:          { type: String, uuid: true, validate: /^[0-9A-F]{32}$/i, },
            name:        { type: String, required: true, }
            status:      { type: String, default: 'active' },
            zip:         { type: String },
        },
        User: {
            pk:          { value: 'account:${accountName}' },
            sk:          { value: 'user:${email}', validate: EmailRegExp },
            id:          { type: String },
            accountName: { type: String },
            email:       { type: String, required: true },
            firstName:   { type: String, required: true },
            lastName:    { type: String, required: true },
            username:    { type: String, required: true },
            role:        { type: String, enum: ['user', 'admin'],
                           required: true, default: 'user' }
            balance:     { type: Number, default: 0 },

            gs1pk:       { value: 'user-email:${email}' },
            gs1sk:       { value: 'user:' },
        }
    }
}

Schemas define your models and their attributes. Keys (pk, gs1pk) can derive their values from other attributes via templating.

Alternatively, you can define models one by one:

const Card = new Model(table, {
    name: 'Card',
    fields: {
        pk: { value: 'card:${number}'}
        number: { type: String },
        ...
    }
})

To create an item:

let account = await Account.create({
    id: '8e7bbe6a-4afc-4117-9218-67081afc935b',
    name: 'Acme Airplanes'
})

This will write the following to DynamoDB:

{
    pk:         'account:8e7bbe6a-4afc-4117-9218-67081afc935b',
    sk:         'account:98034',
    id:         '8e7bbe6a-4afc-4117-9218-67081afc935b',
    name:       'Acme Airplanes',
    status:     'active',
    zip:        98034,
    created:    1610347305510,
    updated:    1610347305510,
}

Get an item:

let account = await Account.get({
    id: '8e7bbe6a-4afc-4117-9218-67081afc935b',
    zip: 98034,
})

which will return:

{
    id:       '8e7bbe6a-4afc-4117-9218-67081afc935b',
    name:     'Acme Airplanes',
    status:   'active',
    zip:      98034,
}

To use a secondary index:

let user = await User.get({email: 'user@example.com'}, {index: 'gs1'})

To find a set of items:

let users = await User.find({accountId: account.id})

let adminUsers = await User.find({accountId: account.id, role: 'admin'})

let adminUsers = await User.find({accountId: account.id}, {
    where: '${balance} > {100.00}'
})

To update an item:

await User.update({id: userId, balance: 50})
await User.update({id: userId}, {add: {balance: 10.00}})

To do a transactional update:

let transaction = {}
await Account.update({id: account.id, status: 'active'}, {transaction})
await User.update({id: user.id, role: 'user'}, {transaction})
await table.transact('write', transaction)

TypeScript

OneTable provides TypeScript type declaration files so that OneTable APIs can be fully type checked.

However, OneTable goes further creates type declarations for your table entities and attributes. TypeScript will catch any invalid schema, entity or entity attribute references.

Using TypeScript dynamic typing, OneTable automatically converts your OneTable schema into fully typed generic Model APIs.

For example:

const schema = {
    models: {
        Account: {
            pk:     { type: String, value: 'account:${name}' },
            name:   { type: String },
        }
    }
}

//  Fully typed Account object based on the schema
type AccountType = Entity<typeof schema.models.Account>

let account: AccountType = {
    name: 'Coyote',        //  OK
    unknown: 42,           //  Error
}

//  Create a model to get/find/update...

let Account = new Model<AccountType>(table, 'Account')

let account = await Account.update({
    name: 'Acme',               //  OK
    unknown: 42,                //  Error
})

account.name = 'Coyote'         //  OK
account.unknown = 42            //  Error

Why OneTable?

DynamoDB is a great NoSQL database that comes with a steep learning curve. Folks migrating from SQL often have a hard time adjusting to the NoSQL paradigm and especially to DynamoDB which offers exceptional scalability but with a fairly low-level API.

The standard DynamoDB API requires a lot of boiler-plate syntax and expressions. This is tedious to use and can unfortunately can be error prone at times. I doubt that creating complex attribute type expressions, key, filter, condition and update expressions are anyone's idea of a good time.

Net/Net: it is not easy to write terse, clear, robust Dynamo code for one-table patterns.

Our goal with OneTable for DynamoDB was to keep all the good parts of DynamoDB and to remove the tedium and provide a more natural, "JavaScripty / TypeScripty" way to interact with DynamoDB without obscuring any of the power of DynamoDB itself.

More?

You can read more in the detailed documentation at:

We also have several pre-built working samples that demonstrate OneTable.

Future

We'll be releasing a OneTable GUI which will be a single-table designer with monitoring and migration and maintenance management. If you have suggestions or ideas for how to improve OneTable please let us know. All feedback, contributions and bug reports are very welcome.

SenseDeep with OneTable

At SenseDeep, we've used the OneTable module extensively with our SenseDeep serverless developer studio. All data is stored in a single DynamoDB table and we extensively use one-table design patterns. We could not be more satisfied with DynamoDB implementation. Our storage and database access costs are insanely low and access/response times are excellent.

Contact

You can contact me (Michael O'Brien) on Twitter at: @mobstream, or email and read myBlog.

To learn more about SenseDeep and how to use our serverless developer studio, please visit https://www.sensedeep.com/.

Comments

{{comment.name}} said ...

{{comment.message}}
{{comment.date}}
Comments Closed

© SenseDeep® LLC. All rights reserved. Privacy Policy and Terms of Use.

Consent

This web site uses cookies to provide you with a better viewing experience. Without cookies, you will not be able to view videos, contact chat or use other site features. By continuing, you are giving your consent to cookies being used.

OK