AlertOps and CloudTrail

AlertOps’ alert management system can be integrated with CloudTrail to receive and respond to critical (predefined status mappings) alarms/alerts through email, SMS, push notification or phone alerts. AlertOps would ensure that the alert would reach the appropriate team by using proper workflows, escalation policies and schedules. Based on your ruleset, incidents can be automatically opened and closed, depending on whether CloudTrail reports a problem or a recovery.

The above scenario and scope for integration is due to the fact that AlertOps has a very flexible and simple API/Webhook configuration feature that can be leveraged with CloudTrail’s auditing and action capabilities.

You should also be aware of CloudWatch and SNS services of AWS for this integration

- Amazon CloudWatch

While CloudTrail generates event history continuously, CloudWatch is used to build log groups to send alerts to external parties (through SNS topics). Amazon CloudWatch is a monitoring service for cloud resources of Amazon Web Services (AWS) and the applications that you run on it. It can be used to collect, track and monitor metrics, log files, set alarms and automatically react to changes in resources of your AWS environment. CloudTrail can send out events to CloudWatch log groups. You can then define metric filters and alarms to be triggered based on those filters. You can also trigger event rules in CloudWatch for events sent from CloudTrail. There is a documentation on CloudWatch Integration with AlertOps.

- Amazon Simple Notification Service (SNS)

It is important to know this service that AWS provides for the purpose of notifications since CloudTrail does not directly send out notifications to an external service in case of an event. Instead, we need to create an SNS topic that will receive the notification and then send out the incident information to an external service (in this case AlertOps). AlertOps must be subscribed to the ‘SNS Topic’ to receive payload from it. Amazon SNS is a pub-sub functionality and provides a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.

AlertOps Inbound Integration

We can define rulesets in AlertOps so that Cl oudTrailcan send out alerts to the AlertOps platform. AlertOps would ensure based on these notifications received, that it would always reach out and assign to the correct person/team by utilizing its escalation policies, schedules, and workflow features.

AlertOps provides Inbound Integrations to integrate with numerous monitoring, chat and ITSM tools. You can configure a inbound integration for AWS CloudTrail Events.

At a high level, the flow looks like the diagram shown below – CloudWatch configured with a log group (which you created in the previous step), receives events from CloudTrail. CloudWatch configured with an alarm for this particular log group trail (for a metric), triggers an alarm that is connected to a SNS topic. This SNS Topic would push the message to the HTTPS endpoint provided by AlertOps. AlertOps must be subscribed to the SNS Topic in order to receive notifications. (We will create an example configuration to illustrate this in the upcoming section)

'To configure an Inbound Integration in AlertOps to receive alerts from CloudTrail,

  • Under 'Integrations' select 'Inbound Integrations', select the category 'API' and then select add 'Add API Integration'

  • There are numerous integration options available in AlertOps, select AWS CloudTrail

  • Once you selected the integration, you can then specify basic settings like the integration name, escalation policy, names of the recipients/groups for which the alerts must be assigned to.

  • Once you click save, the API Integration will be created, and you will be given a unique URL which acts as the access point and needs to be configured at the source (in this case CloudTrail), to send alerts. You can find the integration you just created, and you can give advanced settings and define various configurations for the alerts to be received and processed. For example, you can define when to open and close alerts based on the payload obtained from the API call, filters etc.

  • Make a note of the API URL, which will be used in CloudTrail, so it calls a HTTP POST request to this URL with the body in JSON format containing the alert specific information. AlertOps automatically creates an alert when the status variable (Message^NewStateValue) contains 'ALARM'. The incident will also be closed automatically when the status 'OK/INSUFFICIENT' is received from CloudTrail.

  • You can similarly define URL mappings as you want, owing to the flexibility provided by AlertOps’ OpenAPI/Plug-and-Play integrations. You can provide other filters and match with regex expressions as well. You can also test the generated URL with the sample data provided.

Configuration of CloudTrail/CloudWatch/AWS SNS for the Integration,

Configuration of SNS

  • - Go to Services – select Amazon SNS.

  • - In the left tab – select Topics Create Topic

  • - Select Standard – Give a name to the topic. You can configure other options as you need to.

  • - Once you create the topic, in the left tab – select Subscriptions Create Subscription

  • - In the Topic ARN option, select the name of the topic you just created.

  • - For protocol – select HTTPS, and in the endpoint – paste the API URL which you obtained when you created the inbound integration. You can configure other options as you need to.

  • - Once you create the subscription, go to Topics, select the topic you created – you must have a screen as below,

  • - In the “Subscriptions” section, you will have a status that says, “Pending Confirmation”. This means that AlertOps hasn’t yet subscribed to this topic to receive notifications. (The below screenshot shows “Confirmed”)

  • - To subscribe to the topic, go to your AlertOps dashboard, under integrations go to ‘Inbound Log’.

  • - You should have an entry in the log, from AWS, however there wont be an alert created. Select the message ID detail. In the body section, you should have a field that says, “Subscribe URL” and a link as a value. Copy and paste the link in a new tab and you must get a confirmation template. (You can save it if you want to).

  • - Once you have subscribed to the topic, navigate to your AWS console. Now if you open the Subscriptions under Amazon SNS, you should see a status that says “Confirmed”. You can edit the topic and subscription configurations as and how you want it.

Configuration of CloudWatch Log Group for sending out CloudTrail Events,

  • In this configuration (for an example scenario), we will create a CloudWatch alarm that will be triggered when there are more than 3 failed sign-in attempts to the AWS console.

  • - Go to CloudWatch console, in the left navigation pane, select Logs, and select the log group which you created while creating the “trail”.

  • - Select the log group, and under ‘Actions’ select ‘Create Metric Filter’

  • - In ‘Create Filter Pattern’, enter the following – ‘{ ($.eventName = ConsoleLogin) && ($.errorMessage = "Failed authentication") }’. You can test the pattern if you want to. For different filter pattern techniques, refer to the link in the Reference section. Click ‘Next’

  • - Create the filter name as “ConsoleSignInFailures”

  • - Enable ‘Create New’ metric namespace and give the namespace as ‘CloudTrailMetrics’, Metric name as ‘ConsoleSigninFailureCount’ and Metric Value as ‘1’. Leave others default and click ‘Next’. Review and create the metric filter.

  • Once you create the metric filter, select it, and click “Create Alarm”. Give the settings as per the below screenshot.

  • In the next page, you can configure the SNS topic which you created in the previous section so that a notification will be sent to the AlertOps’ unique API URL when an alarm state is raised. Review and create the alarm!

That’s it! You have created a CloudWatch “trail” that will send out logs to CloudWatch log group, which will trigger an alarm for a particular metric. Try logging in 3 or more times (failing which), this alarm notification will be sent and can be viewed in the ‘Inbound Log’ section in AlertOps Dashboard.

Alert Triggering Information:

AlertOps will automatically create an incident when a new alert is received from CloudTrail when the Message^NewStateValue contains "ALARM".

If an alert with status "ALARM" matches an existing Open Alert, AlertOps will recognize the new alert as a duplicate and ignore the alert.

The alert will be recorded in the Inbound Messages table as “Mapped Appended.”

AlertOps will automatically close the same incident when an alert with Message^NewStateValue contains "OK/INSUFFICIENT".

References

AlertOps Integration Guides

General Restful API Guide

CloudTrail Concepts

AWS SNS

CloudWatch

Metric Filter Syntax

Did this answer your question?