ITOps: Managing Alert Quality

Introduction

• As applications and infrastructure grow, ITOps organizations integrate more monitoring tools, leading to more alerts.
• The increasing volume of alerts decreases quality and usefulness, making it difficult to identify alerts that require attention.
• Many organizations don't have structured processes to regularly assess and modify or retire alerts.
• Alert environments that are left unmanaged can overwhelm incident and alert management workflows.
• An organization receiving 500 monitoring alerts in its first year and experiencing 15% growth would have 12,175 alerts after 10 years.
• Initially, 5% of alerts are noise, increasing to where the majority of alert traffic becomes noise by year 10.
• By 2022, a company that started monitoring in 2010 would have three times as many noisy alerts as actionable ones.
• Most alert data is unactionable noise.

Assessing and Managing Alert Quality

• Organizations must categorize different alert "qualities" to reduce alert noise and improve the alerting environment.
• It is important to differentiate between actionable alerts and alerts that generate noise.
• Low-quality alerts are misconfigured or lack the information to support action by the response team, causing value-less overhead.
• Medium-quality alerts contain the minimum information and context needed for operator action but lack business context, dependencies, or resolution steps
• Should include the configuration item (CI) and the symptom of the problem
• Alerts accumulate until critical and escalated to L1/L2 response teams.
• High-quality alerts meet criteria for high actionability by possessing complete technical and business data.
• High-quality alerts include ownership and routing, business impact, runbooks, dependencies, and enrichment context.
• The desired outcome is intelligent process automation and rapid incident resolution by the appropriate team.

Enrichment is Critical to Creating High-Quality Alerts

• Alerts from monitoring tools often lack operational, topological, or other contextual data.
• Without enriching alerts with metadata, ITOps teams must scan low-quality alerts and use a heuristic approach to determine what to focus on.
• Lack of enrichment complicates tasks such as separating noise from meaningful alerts.
• It also complicates tasks like grouping alerts into incidents, surfacing root causes, and routing incidents or triggering automation.
• IT operations must understand which enrichments improve alert data quality.
• Technical and business context improvements are critical in correlation, prioritization, and automation.
• Technical context supports operator actions for medium quality alerts.
• Monitoring tools lack metadata on the relationships between IT assets and services.
• Enriching alert data with technical context supports operator actions
• Continuous integration (CI) and Continuous deployment (CD) information.
• Detected symptom.
• Problem description
• AIOps platforms use machine learning to group and correlate alerts into incidents based on time, topology, and context.
• This ensures that alerts have the necessary context to prioritize incident response.

Business Context Drives Actionability and High-Quality Alerts

• Alerts enriched with technical context enable the algorithmic addition of business context.
• Incident severity, impacted services, business priority and routing.
• For example, issues that interfere with revenue-generating applications and databases would be labaled high priority and automatically escalated, assigning the correct response teams.
• Other business contexts
• Teams that should be notified.
• Relevant customers.
• What is being impacted.
• Custom tags capture context, sort, filter, visualize, and act on alerts
• Tags include payload data to establish escalation paths and reduce response times by guiding operators.

Strategic Pillars for Improving Alert Quality

• Improving alert quality involves empowering staff to react, route, and remediate effectively.
• Less is more
• Resolve alert and incident clutter to increase quality and deliver actionable insights to improve efficiency and resolution times.
• Context is everything
• Enrich alerts with operational, topological, change and time-based dimensions for effective incident correlation.
• Quality is evolutionary
• Build repeatable processes with key performance indicators (KPIs) for assessment and improvement.

Measuring and Reporting on Alert Quality

• Alerts can be assessed for quality based on contextual information checklists.
• Measuring "actionability" requires connecting correlated alerts with operator actions and measuring outcomes.
• Mean time to detection (MTTD), response, and resolution (MTTR).
• Dashboards and visualizations monitor, tune processes, and optimize incident quality.
• The Sankey diagram displays alerts from tools on the left, high, low, and noisy alerts in the middle, and green bars on the right that show operator action.
• ITOps can optimize alert quality from tools by enriching payloads from low-quality sources.
• Low-volume sources of low-quality alerts may be tool rationalization opportunities that enable coverage with higher-quality alerts.
• Retiring unneeded tools saves on licensing costs.

Best Practices for Building High-Quality Alerts Within ITOps

• Tool configuration and long-term commitment from stakeholders are required
• Degree of cultural shift to emphasize shared value
• Focusing improvements on a domain with low alert quality, and having a high level of technical and business context
• Addressing "low hanging fruit" and adding critical information to existing alerts is key
• Establishing Key Performance Indicators (KPIs) and illustrating improvement through analytics.
• ITOps decision makers must be guided by the business impacts of technology issues, rather than the technology issues themselves
• Alert must include defined and agreed upon business context
• Standardize, measure, and improve incident response workflows across cross-functional teams
• KPIs and business outcomes should be reviewed with stakeholders.
• Alert environment maintenance categorizes, escalates, and resolves alerts in a timely fashion
• Monitoring ensures KPIs are measured correctly when resolving unactioned alerts

Enrichment is the Best-Kept Secret for AIOps Success

• High-quality alerts are fundamental to proactive, efficient, and effective ITOps.
• Alert quality improvement begins with a mindset shift.
• Starts by looking at noisy alerts and defining quality standards with Service Level Agreements (SLAs).
• Enrichment drives the filling of information gaps to reduce alert noise, increase operator efficiency, and build a foundation for actionability.