Scheduled Maintenance Window to Upgrade SenseDeep: November 3rd 11:00 PM and 3:00 AM PST.

DynamoDB for Log Storage

The current release of SenseDeep uses DynamoDB to cache recent log data and preserves the master copy of log data in CloudWatch Logs. DynamoDB is an AWS serverless database that offers almost limitless scalability and can ingest vast amounts of data.

DynamoDB has several features that make it ideal for storing log data. It can scale to accommodate heavy simultaneous updating when a burst of log data needs to be ingested and stored. Time-series log data can also be effectively mapped onto the DynamoDB indexing schema.

DynamoDB is highly reliable and stores 6 replicas of your data, however, this resiliency comes with a cost. For log data, extra copies are not required as a master copy of the log data is always available from CloudWatch logs. The extra replicas do however, impose a significant cost to your AWS bill.

Elastic File System (EFS)

The AWS Elastic File System (EFS) is a simple, fast, serverless, fully managed, elastic file system. It offers up to almost 8 Exabytes of storage with a variety of redundancy configurations.

As a storage medium for log data, it offers extremely low cost with fast storage and retrieval times.

New Log Database

SenseDeep has created a high-performance log database that leverages EFS and offers the following features:

• Ultra low write cost when ingesting log data
• Fast query of time-ranged log data
• Scalable for the largest of sites with huge log data volumes
• Full text searching of log data
• Automated expiry of old log data
• Extremely high write capacity

SenseDeep App

The change to SenseDeep for the new logging engine is entirely “under the hood”. The SenseDeep App user interface is not changing with the exception of removing log data caps from the Cloud > Edit panel.

Rollout

The rollout for the new release is scheduled for Nov 3rd between 11:00 PM and 3:00 AM PST.

IMPORTANT Steps to Upgrade

Following the deployment of the release, it will be necessary for you to recreate your cloud connections in SenseDeep to access log data. Please be aware that your recent log data cache will be cleared, but there is no need to worry, as your historical CloudWatch logs data will remain completely unaffected.

To update your cloud connections, follow these steps:

1. Take note of any SenseDeep alarm configurations and log lifespans. Take screen shots of each alarm and the log list so you can recreate their configuration.
2. For each SenseDeep organization, go to the Cloud List and select your cloud to edit.
3. Remove the cloud by clicking Delete and wait 15 minutes for AWS to fully remove the SenseDeep cloud resources.
4. Check in the AWS console under Cloud Formation for the SenseDeep stack. Wait for the remove to complete. If the delete fails (DELETE_FAILED), wait for the delete to complete and then select the stack and click Delete. Elect to retain resources and check the resources and then click delete.
5. Re-recreate your cloud by clicking Add and complete the CloudFormation template. The template can take up to 10 minutes to complete. Please be patient.
6. Re-add your alarms.
7. Reset any custom log lifespans.

If you have any difficulties, please contact us at support@sensedeep.com.

After updating your cloud connection, SenseDeep will start caching your log data in the new log database. If you require past log data, you can select backfill from the Log Edit page to backfill the SenseDeep log cache from CloudWatch logs.

Architectural Changes

In addition to the new logging engine, the new release splits the SenseDeepWatcher lambda into a separate SenseDeepLogger. Previously the Watcher was responsible for ingesting log data and for synchronizing SenseDeep’s map of your AWS resources.

The SenseDeepLogger is now solely responsible for ingesting log data and for running log alarms to check for any alert conditions in your log data. An AWS EFS must run in a VPC, so the Logger lambda runs in its own private, isolated VPC disconnected from the public internet.

As a result of switching to EFS, SenseDeep no longer needs to impose logging data caps. Due to the capacity and scale of EFS, log data caps are not necessary and will be removed in this release.

Questions

If you have any questions prior to the rollout. Please contact us at support@sensedeep.com.