Reliability of applications is a top concern, according to a recent survey of over 300 on-call practitioners, managers, and executives responsible for incident response in production cloud environments of up to 10,000 nodes. With deep integrations to AWS services, Shoreline’s cloud reliability platform has helped enterprises and cloud unicorns as they scale cloud adoption by increasing the productivity of their cloud operations teams.
Already an AWS Partner, today we are excited to announce that Shoreline is also a launch partner for AWS CloudTrail Lake, allowing Shoreline and AWS customers to use CloudTrail Lake as their single source of truth for auditing events. CloudTrail Lake is a managed security and audit data lake that lets organizations aggregate, immutably store, and query events recorded by CloudTrail. Now, CloudTrail Lake supports ingesting activity logs from Shoreline, and is backed by a 7-year default retention policy to help customers meet compliance requirements.
Over the past year, we have collaborated with our customers and the CloudTrail team to engineer this solution, which addresses a key gap in auditing critical operational activities for cloud operations teams. With this integration, Shoreline commands that are run against Amazon EC2 instances or Amazon EKS clusters will be trackable in CloudTrail Lake for auditing purposes. In CloudTrail Lake, customers will be able to see all the actions performed against resources that are managed by Shoreline, as well as those that are not. This greatly reduces unnecessary context switching for cloud operations and site reliability engineering (SRE) teams that are responding to a production incident.
Architecture of Shoreline and AWS CloudTrail Lake integration
Shoreline orchestrates operations at the edge using lightweight agents that run as containers on AWS EC2 instances and daemon set on EKS clusters. In addition to the benefits of faster issue detection and repair, it is also more secure as credentials don’t leave the environment. When the integration is enabled, all the operational activity performed by the Shoreline agent is streamed to CloudTrail Lake without needing to provide any credentials to Shoreline. Since the agent runs on an EC2 instance, it uses the EC2 IAM role to make the request to the CloudTrail API.
Shoreline Audit showing events in the past day
This integration delivers a single source for complete audit information. Shoreline logs every operation performed on resources managed by Shoreline on behalf of users (as shown above), but without the integration, only AWS API calls will show up in CloudTrail. For example, a linux command executed inside a Kubernetes container in EKS would not. Similarly, operational events that happen outside of Shoreline are not captured in Shoreline Audit. For example, AWS API calls made outside of an EC2 instance such as using the AWS console or a CI/CD tool or a user’s desktop. Once this integration is enabled, information from both Shoreline and CloudTrail is available in a single location.
AWS CloudTrail Lake showing events from Shoreline source
The CloudTrail Lake query above shows how to view events that came from Shoreline, in a similar format to the Shoreline Audit UI. The query below is an example of how to combine events from both CloudTrail and Shoreline. It shows all the events that happened for a particular EC2 instance in one place. Customers can easily analyze the events on their infrastructure using CloudTrail Lake’s query capabilities.
AWS CloudTrail showing events from both CloudTrail and Shoreline for an EC2 instance
The integration from Shoreline to CloudTrail is currently not enabled by default. There are a few one time steps:
- Add CloudTrail PutAuditEvents permission to the EC2 instance IAM policy (so that the Shoreline Agent can call the API to send the events).
- Create a CloudTrail event data store for the Shoreline events with the desired retention period
- Create a CloudTrail ingestion channel dedicated for Shoreline event source
- Edit the Shoreline Agent configuration to enable the integration and provide the ingestion channel name created above
For details of these steps, please refer to the Shoreline CloudTrail Lake integration documentation.
Disabling the integration is straightforward. Simply set the CloudTrail integration flag on the Shoreline Agent to false.
With the AWS CloudTrail Lake and Shoreline integration announced today, we are excited to see customers implement powerful operational automations with greater confidence, safety and security.