How to avoid alert fatigue: using alerting in the right way

‍

Introduction

‍

As I mentioned in some of the previous articles, many businesses are reactive in their day-to-day as they try to monitor their operational events & KPIs via dashboards instead of proactive alerting of relevant business events. However, lots of companies go from one extreme to the other, causing alert fatigue in the organisation, and ultimately a quick way back to the static (but at least not annoying) world of dashboards. As in many situations, balance & the right setup are key. This article will explain how to use alerting in business operations in the right way to capture value & provide peace of mind.  

What is alert fatigue and why is it detrimental?

‍

Alert fatigue is a common problem in businesses that rely on a large number of alerts, many of which are irrelevant to the recipients. In these situations it becomes difficult to efficiently handle and even read through notifications, leading to having most of the alerts simply ignored or "muted."

‍

Alert fatigue can be detrimental to a business in a number of ways. First, it can lead to important business events being overlooked or forgotten, which can result in missed opportunities or operational incidents going unchecked. But most importantly,

‍

it can quickly lead to lack of trust in alerting, resulting in many businesses going back to the ineffective use of static dashboards.

‍

Most of the business operations teams have already tried alerting before but unfortunately with the wrong setup, resulting mostly in the above-mentioned outcome.

How to use alerting to avoid this fatigue happening & get the expected value from them

‍

There are several steps businesses can take to avoid alert fatigue and ensure that they are getting the expected value from their alerts.

‍

• Define the right alerts: To avoid alert fatigue, it is important to be selective about which alerts are set up. This means designing alerts that are relevant to the current key priorities of the business and that are tied to specific business events or KPI changes. It can also be helpful to learn from other companies in the industry, either by talking to customers or reading industry articles, to get a sense of what other companies are monitoring.

‍

• Route alerts to the right people & at the right time: To ensure that alerts are being effectively used, it is important to ensure that they are being sent to the right people and at the right time. This means sending alerts to the individuals or teams responsible for the business event or KPI in question, and using the right channels to deliver the alerts. These can be Slack channels for teams working remotely, Whatsapp numbers for people being on the field or the dedicated task management tools for customer service or sales functions. It is also important to ensure that alerts are delivered in real-time, so that they can be acted upon promptly.

‍

• Set up escalation rules: To avoid overwhelming management with too many alerts, it is important to set up escalation rules that determine when an alert should be escalated to them. This can help to ensure that only the most important and unresolved events are brought to their attention, allowing managers to focus on the ones that are most relevant to their work. Companies can even decide to first simply log relevant business events in a task management system, and send notifications only if they are not resolved within a specific time period.

‍

• Ensure that actions are taken based on alerted events: To capture (and communicate the) value from alerting, it is important to ensure that actions are taken based on them. This means assigning owners to alerted events, so that there is accountability for following up on and resolving them. It can also be helpful to attach playbooks to these events, which outline the steps that should be taken to resolve the underlying issues or opportunities. Finally, it is important to follow up on open events to ensure that they are being addressed in a timely manner.

‍

• Automate resolution whenever possible: To make the most of alerts and avoid alert fatigue, it is important to automate resolution whenever possible. This can include re-running scripts to ensure that an alerted event is still relevant, automating communication with stakeholders (by integrating actions into messaging tools), and automating system-level actions (e.g., via webhooks) to resolve the alert. By automating these processes, businesses can ensure that alerts are being effectively resolved.

‍

• Continuously iterate & improve on alerts: To avoid alert fatigue and ensure that alerts are driving value for the business, it is important to continuously iterate and improve on them. This means analyzing alerts to understand what triggers them, when they are triggered, who they are sent to, and how they are resolved. It also means taking learnings from this analysis to make changes to processes, automate steps, modify alert conditions, set new owners or playbooks, and potentially restructure teams as needed.

What kind of tools are out there that meet the above mentioned criteria?

‍

There are several types of tools that can be used to set up and manage alerts, but each has its own limitations.

‍

BI teams may use SQL-based tools like Apache Airflow, but these require manual setup and may not be easily modified by business teams as they need to request even the smallest modifications from their data colleagues.

‍

Self-serve dashboarding tools like Metabase and Looker offer only basic alerting functionality, and do not have advanced features like the ability to set owners, playbooks, or escalation rules, or to integrate with incident or task management tools.

‍

No-code workflow automation tools like Zapier and Make can be used to set up alerts and automated workflows, but may not have the flexibility or scalability needed for complex operations, and do not offer analytics to help business teams improve their day-to-day operations.

The solution?

‍

The solution to these challenges is an end-to-end monitoring and incident management platform specifically designed for business operations teams.

‍

A platform like this should allow business teams to easily set up and modify alert conditions in a few clicks, and to define notification rules and escalation rules across a range of communication tools and third-party SaaS destinations. It should also create events for alerts in a task management system, allowing their resolution to be transparent, consistent, and best practice. Finally, it should offer analytics and insights to help business teams capture learnings and improve their operations.

‍

We’ve built Flawless with the above in mind, namely to help business teams turn their data into actions through data-driven alerting and automation workflows. If you are interested in learning more, feel free to reach out to us!

‍