New Log Data Engine for SenseDeep
A new log engine has landed for SenseDeep. The new log engine offers enhanced speed and reduced cost of ingesting log data on your AWS bill. The prior SenseDeep implementation used the DynamoDB database to store recent log data. While this can be ideal, DynamoDB imposes a non-trivial cost for writing data and for larger sites, this cost can be significant.
The new engine uses the AWS Elastic File System (EFS) to cache recent log data. EFS is a low-cost, highly scalable, persistent filesystem. SenseDeep has created a log database that uses EFS to cache recent log data and this has greatly reduced the cost of ingesting log data.
With the new log engine, you get the same easy access to your most recent log data, but at a fraction of the price on your AWS bill.
DynamoDB for Log Storage
Previously, SenseDeep used DynamoDB to cache recent log data and preserved the master copy of log data in CloudWatch Logs. DynamoDB is an AWS serverless database that offers almost limitless scalability and can ingest vast amounts of data.
DynamoDB has several features that make it ideal for storing log data. It can scale to accommodate heavy simultaneous updating when a burst of log data needs to be ingested and stored. Time-series log data can also be effectively mapped onto the DynamoDB indexing schema.
DynamoDB is highly reliable and stores 6 replicas of your data. However, this resiliency comes with a cost. For log data, extra copies are not required as a master copy of the log data is always available from CloudWatch logs. The extra replicas do however, impose a significant cost to your AWS bill.
Elastic File System (EFS)
The AWS Elastic File System (EFS) is a simple, fast, serverless, fully managed, elastic file system. It offers up to almost 8 Exabytes of storage with a variety of redundancy configurations.
As a storage medium for log data, it offers extremely low cost with fast storage and retrieval times.
New Log Database
SenseDeep has created a high-performance log database that leverages EFS and offers the following features:
- Ultra low write cost when ingesting log data
- Fast query of time-ranged log data
- Scalable for the largest of sites with huge log data volumes
- Full text searching of log data
- Automated expiry of old log data
- Extremely high write capacity
SenseDeep App
The new SenseDeep logging engine is entirely “under the hood”. The SenseDeep App user interface has not changed with the exception of removing log data caps from the Cloud > Edit panel.
Architectural Changes
In addition to the new logging engine, the SenseDeep now splits the SenseDeepWatcher lambda into a separate SenseDeepLogger. Previously the Watcher was responsible for ingesting log data and for synchronizing SenseDeep’s map of your AWS resources.
The SenseDeepLogger is now solely responsible for ingesting log data and for running log alarms to check for any alert conditions in your log data. An AWS EFS must run in a VPC, so the Logger lambda runs in its own private, isolated VPC disconnected from the public internet.
As a result of switching to EFS, SenseDeep no longer needs to impose logging data caps. Due to the capacity and scale of EFS, log data caps are not necessary and have been removed.
Questions
If you have any questions please contact us at support@sensedeep.com.
Try SenseDeep
Start your free 14 day trial of the SenseDeep Developer Studio.
Messages are moderated.
Your message will be posted shortly.
Your message could not be processed at this time.
Error: {{error}}
Please retry later.
{{comment.name || 'Anon'}} said ...
{{comment.message}}