When an Alert is sent to multiple teams, the default behavior is to end the delivery when the Alert is assigned, and only one User can own the Alert. However, in some cases, the need to ensure each team receives the Alert requires the notifications continue until each team acknowledges. In this article, we will create a Workflow to send additional notifications to the team that has not Acknowledged the Alert.
First, a few words about Acknowledgement in AlertOps. Acknowledge is an optional state for Alerts, alongside Close, Assign, or Escalate. By default, Acknowledging an Alert does not change the activity of an Alert; notifications will continue until it is Assigned or Closed. Acknowledge is useful in cases where multiple Users or Groups are notified of an Alert at the same time, when team protocol allows a User to receive an Alert but not take ownership (Assign).
When a user Acknowledges an Alert, that action is recorded in the Alert detail, but Escalation logic continues, and the Alert will be escalated to the next User at the interval defined in the Escalation Rule. In such a scenario, it would be more effective to simply Escalate the Alert, and so Acknowledge is not a default option and must be deliberately enabled in the Escalation Rule. Acknowledgement can also trigger additional activity, as we will explore here.
By using Workflows, the Acknowledge state can serve as a trigger for further actions, and we will use this to build a Workflow that will send reminder notifications to Users who have not Acknowledged the original notification. In our scenario, two Groups will be notified of a new Alert. Each Group has multiple Users. A User from each Group will need to Acknowledge the Alert to end the notifications. Only the active User from each Group who have not acknowledged the Alert will continue to receive notifications.
To begin, the Integration must have two Recipient Groups. In our scenario, an Infrastructure Team and a Dev Team are configured. Company protocol requires the Infrastructure Team to own and resolve the Alert, but the Dev Team must also Acknowledge, so as to be aware and prepared to assist. If the Infrastructure Team assigns themselves the Alert, notification to the Dev Team will continue until Acknowledged.
The composition of the Groups is flexible, they may have 2 Users, or 20. They may be the same size, they may be different sizes. For the sake of simplicity, however, we will utilize Centralized notification settings in the Escalation Rule, and this requires consistent contact methods for both Groups. User based notification settings can be applied, but complexity increases accordingly.
Create a new Escalation Rule and configure the Integration to use it. An existing Rule can be used, but we will use a new one here. We will use email for this example.
Create an addtional Escalation Rule to define the behavior of the reminders. A Escalation Rule used in a Workflow cannot contain a Workflow, so two rules must be used. We will also use separate options in this rule.
With the Integration, Groups, and Escalation Rules in place, create a new Workflow.
Enter a Name for the Workflow
Select Message for the Type
Check the Scheduled box
Set an appropriate Recurrence Interval
Save the Workflow
We now have fields for Start and Stop Conditions, as well as Actions. Under Match All Conditions for Start Conditions, add the following Conditions:
Set the TimeFromBeginning to define the time you wish to allow for the intial Acknowledgement, before the reminders begin.
Next, set the Stop Condition as follows:
Now, press Add Action, and select SendMessage
Select the Escalation Rule (the second one created above)
Check Send To Original Recipients
Enter the Message Text
Press Save and the Workflow is updated. Enable the Workflow by pressing the button in the top right of the page.
Next, apply the Workflow to the Escalation Rule used in the Integration. On the Workflows tab of the Escalation Rule, press Add Workflow
Select the Workflow you created and press the arrow to move it to the Selected Workflows list (it must be enabled to be visible here)
Press Update and the Workflow is added to the Escalation Rule and the Rule is saved.
Select the Message Options tab in the Escalation Rule, and check the box for Email under Acknowledgement Options. If other contact methods are used, select the appropriate methods. Unselect the options for Assign and Close. This will present the Acknowledge and Escalate buttons in the email.
Select the Others tab in the Escalation Rule. Check the box for One Message Per Recipient. Press Save
In the Escalation Rule for the Workflow, do not select Acknowledgement Options. Also, unselect the options for Assignment, Escalate, and Close. We do not want them to act via the reminder, as the reminders exist in a separate thread. Instead the original message must receive the Acknowledgement to stop the workflow.
When an Alert is triggered by the Integration, the Primary member of each Group will receive an email with action buttons.
Pressing the Acknowledge button will open a browser to an AlertOps page
From here, the User can take Assignment. By removing options to Assign or Close from the original message, the User must either Acknowledge or Escalate. If the Alert is Escalated without Acknowledgement, the next User must Acknowledge, and the reminders will shift to the next User.
Below, we see the Alert Detail with two initial messages, both pending Acknowledgement (PACK)
Next, after multiple reminder messages, both Users have Acknowledged (ACK). Note that after 6:40, only one reminder message goes out until the second User Acknowledges. The Source column indicates the message was from the Workflow.
The Workflow message contains no action buttons, and includes (WF) next to the AlertID, to denote the origin as the Workflow
By selecting One Message Per Recipient, there is only one message to Acknowledge, with all subsequent messages coming from the Workflow. If the Alert is Acknowledged, but not Assigned, the original Escalation path is followed, but no further Acknowledgement is required from that Group. The Alert must still be Assigned, and ultimately Closed.