Mastering SLI Implementation

Implementing Service Level Objectives (SLOs) for Site Reliability Engineering (SRE)

SLOs are key to making data-driven decisions about reliability and are at the core of SRE practices.
Engineers are a scarce resource, and investing in the most important characteristics of the most important services is crucial.
SLOs are a tool to help determine what engineering work to prioritize and are essential for SREs.
SLOs set a target level of reliability for the service’s customers, and user happiness is what matters.
An SLO should be owned by someone in the organization who is empowered to make tradeoffs between feature velocity and reliability.
Service level indicators (SLIs) are indicators of the level of service provided and are used to determine the right SLO number.
SLIs are mostly treated as the ratio of two numbers: the number of good events divided by the total number of events.
A small number of SLI types that represent the most critical functionality to customers should be chosen.
SLIs should be specific and measurable, and SLI implementations should require a minimum of engineering work.
SLIs can use various sources, such as load balancer monitoring or application server logs, and can be calculated using Prometheus notation.
SLOs can be defined over various time intervals and can use either a rolling window or a calendar-aligned window.
The most important goal is to get something in place and measured, and to set up a feedback loop for continuous improvement of SLO targets.Establishing and Managing Service Level Objectives (SLOs)
SLOs are a way to measure the reliability of a service based on specific performance indicators called Service Level Indicators (SLIs).
SLIs should be chosen based on their relevance to the user experience, and the SLOs should be agreed upon by all stakeholders.
SLOs can be defined for a specific period of time, such as a rolling four-week window, or aligned with business planning and project work, such as quarterly evaluations.
Once SLOs are established, an error budget policy should be defined to determine what actions to take when the service exceeds its error budget.
Continuous improvement of SLO targets is necessary, which can be achieved by measuring user satisfaction and identifying critical user journeys.
Bucketing can be used to distinguish certain classes of requests and apply different SLOs to them.
SLOs can also be used to coordinate and implement reliability requirements between different components in a system.
Dashboards and reports can be used to monitor SLO compliance and identify problematic areas.
The appropriate course of action when the error budget is exhausted should be covered by the error budget policy and agreed upon by all stakeholders.
Decision making using SLOs and error budgets can include prioritizing reliability-related bugs, stopping feature launches, or declaring an emergency with high-level approval.
SLOs should be documented in a prominent location for review by other teams and stakeholders, and the error budget policy should also be documented.
Enforcing an error budget policy involves taking specific actions when the service has consumed its entire error budget, and common actions include stopping feature launches or devoting engineering time to working on reliability-related bugs.Managing Reliability with Service Level Objectives (SLOs)
SLOs help communicate the level of reliability a service can provide to users.
Inherent reliability limitations should be acknowledged in SLOs, and engineers should engineer around those limitations.
Deploying a service in two zones to increase availability assumes independence between instances, which is almost never the case.
An error budget policy should address missed SLOs caused by dependencies handled by another team.
Freezing changes may not be practical in addressing missed SLOs, so decide what is most appropriate for the service and its dependencies.
Experimenting with relaxing SLOs should be done carefully and only if there is enough error budget to burn.
The relationship between a measurable technical metric and a key business metric can be identified through experimentation.
Regularly reviewing the validity of the relationship between technical and business metrics is important.
Misinterpreting data from experiments can lead to incorrect conclusions about SLOs and user behavior.
SLOs offer a framework to discuss system behavior with greater clarity and can help pinpoint actionable remedies when services fail to meet expectations.
Appendixes in the book provide examples of SLO documents and error budget policies.
SLOs should be regularly reviewed and updated as the service evolves and user expectations change.