If you are using AWS Lambda or Serverless and you need to trace and debug requests across multiple services, then SenseDeep correlated logs may be the solution for you.
Serverless applications often have multiple Lambda functions or services that cooperate to respond to a single client request. Requests that originate with a single client request, may traverse through API Gateway, one more more Lambda functions, Kinesis streams, SQS queues, SNS messages, EventBridge events and other AWS services. The request may fan out to multiple Lambdas and results may be combined into a single response.
Unfortunately, this means that the log data for a single request is scattered over multiple AWS CloudWatch log groups.
Consequently, diagnosing a single request can be like searching for a needle in a haystack. Tracing a request over multiple log groups can be difficult and time consuming.
SenseDeep addresses this issue via Meta logs that combine multiple AWS CloudWatch log groups in real-time into a unified correlated log view.
You can search for a single request using CloudWatch, however this method has some key limitations.
For example, you can use the following CloudWatch Insights query to retrieve log events from multiple log groups. First you select each of the logs for the query, then enter the query command.
fields @timestamp, @log, @message, x-correlation-id | filter @message like /MENkHjqOIAMESfg=/
This extracts the log messages that contain the given message pattern string. This manual method works, is slow to setup, isn't very scalable and has a few major limitations.
First and foremost, it can be slow, ... very slow. CloudWatch insights often takes 20+ seconds to fetch results and can take up to 15 minutes if the event you are searching for did not happen very recently.
Second, after finding the matching log events, you cannot see the logs just before or after the event. The root cause of a request failure may be in log events that happened just prior to the request. But you cannot easily navigate to see those events.
Third, the logs must be in the same account and region. You cannot correlate requests across different AWS accounts or regions. If your services are delivered from different AWS accounts, you are out of luck.
SenseDeep addresses these issues via Meta Logs.
A SenseDeep meta log is a correlated view over multiple CloudWatch log groups. From a meta log, you can view log events ordered in sequence regardless of the AWS service or log group. This log view behaves like any other log view, except that it is combining log groups and streams from multiple sources (even cross accounts or regions).
From the meta log, you can immediately locate and isolate any specific request by searching for a request ID or pattern of your choosing to isolate the complete request trace.
Here is a sample view of a request that flows through two lambdas, an EventBridge bus and a final lambda. The events have been filtered by a correlation ID.
Creating Meta Logs
To create a meta log, click on "Logs" and then click "Add". Enter your meta log name and select the logs to combine.
You can explicitly select the individual contributing logs entering a regular expression pattern that will be used to dynamically match contributing logs. Using a regular expression pattern is preferable if you have a changing set of log group names.
Alternatively, you can select specific logs via the log list combo box.
High Cardinality IDs
To get the most out of meta logs, you should utilize a unique request ID that is passed to all your Lambda functions and then emitted in all log events. This is sometimes called a High Cardinality ID. Using this ID, you can filter log events using this ID and display only those events for that request ID. Two clicks on a request ID in the viewer will open the View panel with the matching pattern set to the request ID. Click OK and the viewer will display the desired request log details.
For effective serverless development and debugging, you need to be able to quickly correlate and isolate requests across multiple cooperating AWS services such as API Gateway, Lambda, SQS and SNS. SenseDeep provides a real-time, fast meta log facility to create a unified view of a request as it passes through multiple AWS services.
Try the SenseDeep DynamoDB studio with a free developer license at SenseDeep App.
Start your free 14 day trial of the SenseDeep Developer Studio.