DynamoDB Attribute Packing for Single Table Designs

A DynamoDB secondary index can select which data item attributes to project (replicate) to the index. It can project all item attributes, a subset of the attributes or only the key attributes.

If you project only the keys, then a read from the secondary index will return the key attributes. With these keys you can read all the remaining attributes from the primary index, but that will incur an additional read request and require code to manage the second request.

With DynamoDB, choosing which attributes to project to secondary indexes can be a challenge. With DynamoDB single table designs, you store multiple entities with different named attributes in a single table. This means the set of attributes to project to GSIs may be large and diverse.

Furthermore, once defined, you cannot change the names of projected attributes after you create the GSI. These issues can make efficient use GSIs difficult and evolving and changing your data design problematic.

OneTable solves this problem by supporting the mapping and packing entity attributes into a single GSI attribute. This makes the task of defining which attributes to project to the GSI relatively easy and also permits changing your data design without having to recreate the GSIs.

OneTable Attribute Mapping

OneTable schemas can define an attribute mapping via the map schema property. This defines the attribute name used in the physical table.

{
    User: {
        email: { type: String, map: 'data' }
    }
}

This will store the User.name property in the data table attribute.

One to One Mapping

OneTable mapping definitions can also map multiple different entities onto the same attribute name.

{
    Account: {
        name: { type: String, map: 'data' }
    }
    User: {
        email: { type: String, map: 'data' }
    }
}

This will store both the Account.name and User.email values in the GSI ‘data’ attribute depending on the data item.

Many to One Packing

Sometimes, you may need to project multiple field properties into a GSI. By using OneTable mappings, you can map and pack multiple attributes from a single entity to a single GSI attribute.

By specifying a mapped name that contains the period character, you can pack property values into an object stored in a single attribute. OneTable will transparently pack and unpack values on read/write operations.

const Schema = {
    version: '0.1.0',
    format: 'onetable:1.0.0',
    models: {
        User: {
            pk:          { value: 'user:${email}' },
            sk:          { value: 'user' },
            id:          { type: String },
            email:       { type: String, map: 'data.email' },
            firstName:   { type: String, map: 'data.first' },
            lastName:    { type: String, map: 'data.last' },
        }
    }
}

This will pack the User.email, User.firstName and User.lastName properties under the GSI data attribute. The data attribute will have the values:

{
    email: 'email-name',
    first: 'firstName',
    last: 'lastName',
}

You can also map and pack the properties from multiple entities into a single attribute name.

By using the map facility, you can create a single GSI data attribute that contains all the required attributes for access patterns that use the GSI. By modifying the OneTable schema and using the OneTable CLI for migrations, you can easily evolve your design without recreating your GSIs.

Using Mapped Attributes

When issuing APIs that write to a mapped attribute, you must provide all the properties that map to that attribute for the entity.

For example, the following will fail because the lastName is not provided and the API must provide all three properties: email, firstName and lastName that map to the data attribute.

await User.update({email: 'coyote@acme.com', firstName: 'Peter'})

Value Templates

There is one other technique you can use for one-way attribute packing.

A OneTable schema field can define a value property which operates like a JavaScript template string. Embedded ${field} references are expanded to create the attribute value. For example:

{
    sk:      { type: String, value: 'user:${country}:${zip}:${state}:${address}' },
    country: { type: String },
    zip:     { type: String },
    ...
}

This will create a sort-key (sk) attribute with the values of country, zip, state and address catenated after a user: prefix. This is useful for queries that can search on varying segments of the sort key using begins_with.

Note: this technique replicates the attributes in the value template and is thus not a technique to reduce overall data storage.

OneTable Follow

If reading from a secondary index that projects a subset of attributes and you wish to fetch the entire item, you would normally have to issue a second read to fetch the full item from the primary index.

OneTable makes this easier by using the follow option where OneTable will transparently follow the retrieved primary keys from a GSI and fetch the full item from the primary index so that you do not have to issue the second read manually.

let account = await Account.find({name: 'acme'}, {index: 'gs1', follow: true})

Under the hood, OneTable is still performing two reads to retrieve the item but your code is much cleaner. For situations where the storage costs are a concern, this approach allows minimal cost, keys-only secondary indexes to be used without the complexity of multiple requests in your code.

We also have several pre-built working samples that demonstrate OneTable and attribute packing.

• OneTable Packed Attributes Sample — How to use packed attributes.
• OneTable TypeScript Sample — Basic TypeScript use.
• OneTable Overview Sample — A quick tour through OneTable.
• OneTable CRUD Sample — Basic CRUD.
• OneTable Migrate Sample — how to use OneTable Migrate and the Migrate CLI.
• OneTable SenseDeep Sample — How to access SenseDeep log data.
• All OneTable Samples

Links

• OneTable Post
• OneTable Migrate CLI
• DynamoDB Single Table Design
• OneTable Migrate CLI
• SenseDeep App