DynamoDB with OneTable Schemas

DynamoDB is a key-value and document database that does not enforce a schema for your data. You can store data items where each item may have different attributes and attribute types. Item values may be primitive values, scalars or compound documents.

This offers great flexibility but with DynamoDB single-table designs, different types of items (entities) will have their own unique set of attributes. It is helpful to define the “schema” of these entities so you can centrally manage the entity signatures and reliably store and retrieve items of different types.

The DynamoDB OneTable Library enables you to define your single-table entity definitions via a OneTable schema. This makes understanding and working with single-table designs dramatically easier and allows another layer of capabilities over the raw DynamoDB engine.

You can also use the SenseDeep Single Table Designer to help you create your OneTable schema via the GUI schema builder.

OneTable Schema

When using OneTable, you define your your entities (models), keys, indexes and attributes via a OneTable Schema. This defines the possible entities and all their attributes. Attributes are specified with their type and other properties for OneTable to validate and control the attributes.

For example, a OneTable schema for two entities and two indexes could look like this:

const MySchema = {
    version: '0.1.0',
    format: 'onetable:1.0.0',
    indexes: {
        primary: { hash: 'pk', sort: 'sk' }
        gs1:     { hash: 'gs1pk', sort: 'gs1sk' }
        gs2:     { hash: 'gs2pk', sort: 'gs2sk', follow: true }
    },
    models: {
        Account: {
            pk:          { value: 'account:${name}' },
            sk:          { value: 'account:' },
            id:          { type: String, uuid: true, validate: /^[0-9A-F]{32}$/i },
            name:        { type: String, required: true }
            status:      { type: String, default: 'active' },
            zip:         { type: String },
        },
        User: {
            pk:          { value: 'account:${accountName}' },
            sk:          { value: 'user:${email}', validate: EmailRegExp },
            id:          { type: String, required: true },
            accountName: { type: String, required: true },
            email:       { type: String, required: true },
            firstName:   { type: String, required: true },
            lastName:    { type: String, required: true },
            username:    { type: String, required: true },
            role:        { type: String, enum: ['user', 'admin'],
                           required: true, default: 'user' }
            balance:     { type: Number, default: 0 },

            gs1pk:       { value: 'user-email:${email}' },
            gs1sk:       { value: 'user:' },
        }
    }
}

Schema Benefits

There are alternatives to using a centralized schema such as implicit typing offered by TypeScript. But the OneTable schema goes beyond simple typing and adds capabilities for validations, default values, generated UUIDs, required and unique attributes, mapped aliases and formatting of values.

Attribute Filtering

OneTable will filter attributes as they are written to your DynamoDB table. If you write an entity that has other additional (superflous) properties, OneTable will only write the attributes defined in the schema.

Similarly, if the table has legacy (unused) attributes in the table, OneTable will filter these before returning items. If the a query specifies the params.hidden property, these “unknown” attributes can be returned.

This filtering enables you to store and manage additional state in your application entity objects without committing these to your database table.

Attribute Validation

OneTable schema attributes can specify a validation regular expression that must match an attribute value before a write to the database will succeed. By defining validation routines on all relevant attributes, you can ensure a higher level of data integrity in your table.

{
    id:     { type: String, uuid: true, validate: /^[0-9A-F]{32}$/i },
    format: { type: String, validate: /^(json)|(text)$/ },
}

Enumerations

Another way to validate an attribute value before writing to the database is to specify a set of values that the item may take. By defining an enum property, you can specify a list of valid values for the attribute.

OneTable will ensure the attribute has a value from the enumerated set of valid values. Note: this is distinct from the DynamoDB set data type. You can use either the OneTable or DynamoDB capability to achieve the same result.

{
    severity:       { type: String, enum: ['critical', 'error', 'warning', 'info'] },
}

Template attributes

It is good practice to uncouple your physical key values from your item attributes. This gives you greater freedom to change your entity signatures and evolve your data designs going forward.

OneTable supports this practice via string templates that define an attributes value. The OneTable value schema property defines the value to be written to the table based on the values of other attributes.

The value property defines a literal string template, similar to JavaScript string templates, that is used to compute the attribute value. The template string may contain ${name} references to other entity attributes. This is useful for computing key values from other attributes and for creating compound (composite) sort keys.

{
    pk:    { value: 'account:${name}' },
    sk:    { value: 'account:' },
    name:  { type: String, required: true }
}

Generated IDs

When you have a need for a generated ID, you can select from a suite of OneTable ID generators.

• UUID — fast non-cryptographic UUID string.
• KSUID — time-based sortable, unique sequential number.
• ULIDs — time-based sortable, unique sequential number.

You can select your ID generator via the uuid, ksuid or ulid schema property.

{
    id: { type: String, ulid: true }
}

Note: to use the KSUID, you need to provide your own KSUID implementation via the Table constructor.

Required Attributes

Entities often have required attributes that must always be defined.

You can specify that an attribute is required and must be provided or defined when creating an item via the required property.

{
    name: { type: String, required: true }
}

You should set “required: true” for all mandatory attributes. Note: this does not impact the properties provided when updating an existing item.

Default Attributes

When some entities are created, certain attributes typically take default values. You can define the default value for an attribute by setting the default property. This value will be used when creating an item for which the attribute is not defined.

The value can be set to either a primitive value or a function that will return the default value.

{
    status: { type: String, default: 'active' }
    plan:   { type: String, default: (model, fieldName, item) => {
            if (item.status == 'active') {
                return 'pro'
            } else {
                return 'free'
            }
        }
    }
}

When a function is specified, the value can be determined based on the other item attributes or other application state.

Formatted Values

Sometimes the values you store in the database are best written in a different format. A classic example is dates where you may wish to store dates as unix epoch number of seconds since 1970, but in your application you wish to interact with JavaScript Date object values. While OneTable does this automatically for dates, you may have other entity attributes that need similar decoding and formatting on reading and writing.

Schema attributes can specify a formatter function that will be invoked to transform the data on all reads and writes.

{
    role: {type: String, transform: (model, op, fieldName, item) => {
        if (op == 'read') {
            //  Convert the attribute to an application role object
            return new Role(item[fieldName])
        } else if (op == 'write') {
            //  Convert the role object to a string representation
            return item[fieldName].toString()
        }
    }}
}

Mapped Attributes

To reduce the total size of the data stored, you can define an alias for an attribute name using the map property. When storing the data in the DynamoDB table, the mapped name will be used. For example, the attribute accountName could be mapped to the shorter abbreviation of “an” and thus reduce the storage required for each item.

OneTable will automatically save items using the mapped name and will convert back to the full attribute name when retrieving items.

{
    accountName: { type: String, map: 'an' }
}

Packed Attributes

Sometimes, you may need to project multiple entity properties into a single GSI. By using OneTable mappings, you can map and pack multiple attributes from multiple entities to a single GSI attribute.

By specifying a mapped name that contains the period character, you can pack property values into an object stored in a single attribute. OneTable will transparently pack and unpack values on read/write operations.

const Schema = {
    version: '0.1.0',
    format: 'onetable:1.0.0',
    models: {
        User: {
            pk:          { value: 'user:${email}' },
            sk:          { value: 'user' },
            id:          { type: String },
            email:       { type: String, map: 'data.email' },
            firstName:   { type: String, map: 'data.first' },
            lastName:    { type: String, map: 'data.last' },
        }
    }
}

This will pack the User.email, User.firstName and User.lastName properties under the GSI data attribute.

By using the map facility, you can thus create a single GSI data attribute that contains all the required attributes for access patterns that use the GSI.

Unique properties

DynamoDB does not provide any native capability for ensuring a non-key attribute is unique. However you can simulate this by performing a transaction which wraps creating the desired item and creating a special unique key for the attribute.

OneTable implements this technique via the unique property automates this pattern.

{
    token: { type: String, unique: true }
}

Via the unique property, OneTable will create a special item with the primary key set to _unique:Model:Attribute:Value. The original item and the unique item will be created in a transparent transaction. The item will be created only if the unique attribute is truly unique.

The remove API will appropriately remove the special unique item.

Indexes

To make using keys-only secondary indexes easier, OneTable has a helpful follow option that will return a complete item from a secondary index.

If reading from a sparse secondary index that projects keys only, you would normally have to issue a second read to fetch the full attributes from the desired item. By using the follow option, OneTable will transparently follow the retrieved primary keys and fetch the full item from the primary index so that you do not have to issue the second read manually.

You can specify the follow option on the schema index, or you can specify on a per-API basis.

{
    indexes: {
        primary: { hash: 'pk', sort: 'sk' }
        gs1:     { hash: 'gs1pk', sort: 'gs1sk', follow: true }
    }
}

This will cause all access via the gs1 index to transparently use the retrieved keys and fetch the full item from the primary index.

Alternatively, you can add the follow option when using the get or find APIs.

let account = await Account.find({name: 'acme'}, {index: 'gs1', follow: true})

Under the hood, OneTable is still performing two reads to retrieve the item but your code is much cleaner. For situations where the storage costs are a concern, this approach allows minimal cost, keys-only secondary indexes to be used without the complexity of multiple requests in your code.

Schema Attribute Properties

The following is the full set of supported OneTable schema attribute properties.

Read more at DynamoDB OneTable.

We also have several pre-built working samples that demonstrate OneTable.

• OneTable Overview Sample — A quick tour through OneTable.
• OneTable CRUD Sample — Basic CRUD.
• OneTable TypeScript Sample — Basic TypeScript use.
• OneTable Migrate Sample — how to use OneTable Migrate and the Migrate CLI.
• OneTable Packed Attributes Sample — How to use packed attributes.
• OneTable SenseDeep Sample — How to access SenseDeep log data.
• All OneTable Samples

Links

• SenseDeep DynamoDB Studio
• OneTable Post
• OneTable Migrate CLI
• DynamoDB Checklist
• DynamoDB Single Table Design
• DynamoDB Sparse Indexes
• OneTable Migrate CLI
• SenseDeep App