Tech — Inside SenseDeep
This is an overview of the SenseDeep service and how it works. For the impatient, here is a short version that describes only the features: SenseDeep Data Sheet.
What SenseDeep Does
SenseDeep performs three key tasks:
Identify vulnerabilities in your cloud configuration and server instances and helps you eliminate those vulnerabilities.
Detect attacks and compromises of your site in real-time and automatically invokes defenses.
Capture and record event history and log files for future forensic analysis if required.
SenseDeep does this by verifying your cloud and server configuration in real-time and looking for vulnerabilities. SenseDeep constantly monitors your site for changes that indicate attacks or compromises and then invokes defenses and notifies you if required.
SenseDeep Security Service
SenseDeep manages your AWS site and any on-premises servers you may have. It uses deep hooks into many AWS services in order to provide real-time automated monitoring of your site and AWS services. SenseDeep is integrated with: AWS VPC, IAM, EC2, AutoScale, RDS including Aurora, IAM, CloudWatch Events, CloudWatch Logs, Lambda, SNS, and S3.
To grant and control access to your account, SenseDeep creates an IAM role and policy in your AWS account. The AWS role specifies SenseDeep as a trusted entity with a unique external ID. This ensures that only SenseDeep and no other party can access your account.
SenseDeep App Portal
The SenseDeep app is the primary user interface to SenseDeep. It provides a security status dashboard and manager interface.
SenseDeep aggregates the security status for each service and server and provides an overall account security status. This is presented as an Attacks gauge that indicates if you are being attacked or have been attacked and compromised. The Threats gauge represents latent threats and indicates if you are vulnerable to future attacks in any of your services. These status gauges are automatically updated in real-time in response to any changes on your site.
The SenseDeep App is also your interface to manage the security of your site. If you wish, you can customize which servers are managed, what regions should be considered and which threat rule packages are utilized.
Microservices Under the Hood
Internally SenseDeep runs several microservices:
- SenseDeep Watch — receives real-time AWS CloudWatch events.
- SenseDeep Audit — performs scheduled and on-demand account configuration audits.
- SenseDeep Admin — performs account and database maintenance.
- SenseDeep Agent Manager — manages the server agents.
The SenseDeep Watch service receives AWS CloudWatch events and determines which parts of the AWS service are impacted. If required, it then invokes the SenseDeep Audit service to check the new configuration and recalculate the security status.
For example: if your site uses AWS AutoScale, scale events are received by SenseDeep and your newly created instances are automatically incorporated for management by SenseDeep. Upon termination of a server instance, a terminate event is similarly received by SenseDeep and the instance is excluded from further management. During this process, there are no false alerts or notifications. We believe in silence when everything is operating as it should.
The audit microservice checks your current AWS account and services for configuration and security vulnerabilities.
The audit service is invoked to audit an account whenever any change is made. Changes trigger an audit of the affected service only. In this manner, you get real-time updated security status without waiting for the "daily scan" of your account that many other security services require. This reduces the window of opportunity for attackers.
SenseDeep Agent Manager
The SenseDeep Agent is an optional host-based intrusion detection and prevention agent. It is installed on your AWS EC2 instances or on your on-premises servers. The Agent will detect any changes to the infrastructure in real-time.
The Agent manager supervises servers that are running the SenseDeep agent. The manager communicates with the agents, creates the threat detector rule sets and receives incoming threat alerts from the agents.
When an incoming alert is received from an agent, the manager recomputes the new security status for the server, service and account and then invokes required defensive countermeasures. These defenses may include stopping or terminating the server, running a Lambda function or web hook and sending notifications.
The manager maintains a communications tether to each agent to ensure the agent cannot be compromised, replaced or modified, in any way, without being detected.
The SenseDeep agent provides log capture, intrusion detection and prevention and vulnerability scanning.
Provisioning the Agent
The agent can be baked into an AMI image or it can be provisioned as part of the EC2 instance user-data initialization script. To register an agent, a SenseDeep server token is specified during installation. These tokens can be regenerated and old tokens expired via the SenseDeep App. Once installed, the agent acquires a unique auth token for its exclusive use that is used to authenticate the server and to secure all communications.
The agent does not open any listening ports and only communicates outbound with the SenseDeep service. All communications with SenseDeep are encrypted using TLS and a cryptographic session identifier.
If the agent is attacked or agent communications with the service are disrupted for any reason, the service will raise an alert. For example: a "kill -9" on the agent will immediately flag the server as compromised.
When the agent starts and at regular intervals thereafter, the agent computes a configuration fingerprint of all installed software on the server. This is sent to the SenseDeep service and a corresponding tailored security rule set is downloaded. The rules set directs the agent to look for specific relevant threats. If the configuration of the server changes, the rule set is updated automatically.
The rule set, like other critical system files is monitored for tampering.
Via the downloaded rule set, the agent will detect threats including:
- probes of network ports
- probes of http web servers
- probes of account logins
- modifications to critical system files
- execution of unexpected processes
- unauthorized login attempts
- attempts to circumvent security
- denial of service attacks
The agent uses highly efficient Linux O/S APIs such as inotify and the Systemd Journal API to monitor changes in the system. The BPF is also used to perform efficient in-kernel network traffic analysis. These mechanisms use event based algorithms with minimal polling to detect most threats in real-time. Other checking mechanisms are also used in cases where these event mechanisms can be bypassed by attackers to ensure robust detection of threats.
When a threat or compromise is detected, the agent will send an alert report to the SenseDeep service. The agent will capture the full threat context and correlate with other coincident alerts. It will employ DoS protection strategies to reduce the impact of a cascade of alerts.
The agent may be configured to invoke local automated defenses in response to threats. These include modifying firewall rules to block attackers, terminating rogue processes or running a local command.
The agent captures host log file data and events and stores them automatically in Cloud Watch Logs for later forensic analysis. By default, the agent will capture all system events via the SystemD Journal log using a high performance data collector. Additional log files can be configured for capture via the SenseDeep App.
The SenseDeep agent is an extremely small and efficient real-time security management agent. It consumes less than 1% of the CPU and is less than 4MB in memory footprint. This is less than 10% the size of most other log capture and intrusion detection agents.
The agent uses the Safe Embedded Runtime for secure and efficient platform services. This runtime has been proven over more than a decade of use in secure embedded systems. It includes services for safe memory allocation, buffer and string handling, parsing and communications.
The Secure Web Site Case Study digs deeper and looks at how SenseDeep is securely implemented.