Google SRE Masterclass 2024

SRE Part 1

https://rsvp.withgoogle.com/events/sre-masterclass-with-google-p1-hel

Introduction

https://sre.google/sre-book/introduction/

The Sysadmin Approach to Service Management

Historically companies have employed systems administrators to run complex systems
Involves assembling existing software components and deploying them to work together to produce a service.
As system grows in complexity and traffic volume, the sysadmin team grows to absorb the additional work.
Pros:
- Relatively easy to implement: as a familiar industry paradigm, there are many examples from which to learn.
- Relevant talent pool is already widely available.
- An array of existing tools, software components, and integration companies are available to help run those assembled systems, so a novice sysadmin team doesn’t have to reinvent the wheel and design a system from scratch.
Cons:
- This approach and the accompanying development/ops split has a number of disadvantages and pitfalls. These fall broadly into two categories: direct costs and indirect costs.
  - Direct costs are neither subtle nor ambiguous. Running a service with a team that relies on manual intervention for both change management and event handling becomes expensive as the service and/or traffic to the service grows, because the size of the team necessarily scales with the load generated by the system.
  - Indirect costs of the development/ops split can be subtle, but are often more expensive to the organization than the direct costs.
  - These costs accrue from:
    - The two teams are quite different in background
    - Skill set
    - Incentives.
    - Use different vocabulary to describe situations
    - Carry different assumptions about both risk and possibilities for technical solutions
    - Have different assumptions about the target level of product stability.
  - The split between the groups can easily become one of not just incentives, but also communication, goals, and eventually, trust and respect. This outcome is a pathology.
  - Traditional operations teams and their counterparts in product development thus often end up in conflict, most visibly over how quickly software can be released to production.
    - Development teams want to launch new features and see them adopted by users.
    - Ops teams want to make sure the service doesn’t break while they are holding the pager. Because most outages are caused by some kind of change—a new configuration, a new feature launch, or a new type of user traffic—the two teams’ goals are fundamentally in tension.

Google’s Approach to Service Management: Site Reliability Engineering

Hiring

Google’s SRE teams focus on hiring software engineers to run their products and to create systems to accomplish the work that would otherwise be performed, often manually, by sysadmins.
The result of their approach to hiring for SRE is that they end up with a team of people who:
- Will quickly become bored by performing tasks by hand
- Have the skill set necessary to write software to replace their previously manual work.
Design and implement automation with software to replace human labor.
Without constant engineering, operations load increases and teams will need more people just to keep pace with the workload.
- Eventually scales linearly with service size: if the products supported by the service succeed, the operational load will grow with traffic. That means hiring more people to do the same tasks over and over again.
- To avoid this fate, the team tasked with managing a service needs to code or it will drown.
Google caps all manual labor at 50% of allocated work time to ensure engineers have enough time to automate systems
- Pros: SREs are directly modifying code in their pursuit of making Google’s systems run themselves.
- Such teams are relatively inexpensive—supporting the same service with an ops-oriented team would require a significantly larger number of people.
- Number of SREs needed to run, maintain, and improve a system scales sublinearly with the size of the system.
- Cons:Competes for the same candidates as the product development hiring pipeline
- Hiring bar high in terms of both coding and system engineering skills means that their hiring pool is necessarily small.
- Not much industry information exists on how to build and manage an SRE team
- Once an SRE team is in place, their approaches to service management require strong management support.
  - For example, the decision to stop releases for the remainder of the quarter once an error budget is depleted might not be embraced by a product development team unless mandated by their management.

Tenets of SRE

SRE team is responsible for the availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning of their service(s).

Ensuring a Durable Focus on Engineering

Google caps operational work for SREs at 50% of their time.
Their remaining time should be spent using their coding skills on project work.
In practice, this is accomplished by monitoring the amount of operational work being done by SREs, and redirecting excess operational work to the product development teams:
- Reassigning bugs and tickets to development managers
- [Re]integrating developers into on-call pager rotations, and so on.
  - The redirection ends when the operational load drops back to 50% or lower.
  - This also provides an effective feedback mechanism, guiding developers to build systems that don’t need manual intervention.
  - Works well when the entire organization—SRE and development alike—understands why the safety valve mechanism exists, and supports the goal of having no overflow events because the product doesn’t generate enough operational load to require it.
Postmortems should be written for all significant incidents.
Postmortems that did not trigger a page are even more valuable, as they likely point to clear monitoring gaps.
Postmortem should establish:
- What happened in detail
- Find all root causes of the event
- Assign actions to correct the problem or improve how it is addressed next time.

Pursuing Maximum Change Velocity Without Violating a Service’s SLO

100% is the wrong reliability target for basically everything (pacemakers and anti-lock brakes being notable exceptions).
100% is not the right reliability target because no user can tell the difference between a system being 100% available and 99.999% available.
- There are many other systems in the path between user and service (their laptop, their home WiFi, their ISP, the power grid…)
- Those systems collectively are far less than 99.999% available.
  - Thus, the marginal difference between 99.999% and 100% gets lost in the noise of other unavailability
    - User receives no benefit from the enormous effort required to add that last 0.001% of availability.
What is the right reliability target for the system?
- Not a technical question: it’s a product question, which should take the following considerations into account:
  - What level of availability will the users be happy with?
  - What alternatives are available to users who are dissatisfied with the product’s availability?
  - What happens to users’ usage of the product at different availability levels?
- The business or the product must establish the system’s availability target.
- Once target is established, the error budget is one minus the availability target. A service that’s 99.99% available is 0.01% unavailable. That permitted 0.01% unavailability is the service’s error budget. We can spend the budget on anything we want, as long as we don’t overspend it.
Use of an error budget resolves the structural conflict of incentives between development and SRE.
SRE’s goal is no longer "zero outages"
Aim to spend the error budget getting maximum feature velocity. This change makes all the difference. An outage is no longer a "bad" thing—it is an expected part of the process of innovation, and an occurrence that both development and SRE teams manage rather than fear.

Monitoring

One of the primary means by which service owners keep track of a system’s health and availability.
Monitoring strategy should be constructed thoughtfully.
Common approach to monitoring is to watch for a specific value or condition and trigger an email alert when that value is exceeded or that condition occurs.
- This type of email alerting is not an effective solution
- System that requires a human to read an email and decide whether or not some type of action needs to be taken in response is fundamentally flawed.
- Monitoring should never require a human to interpret any part of the alerting domain. Instead, software should do the interpreting, and humans should be notified only when they need to take action.

Alerts

Signify that a human needs to take action immediately in response to something that is either happening or about to happen, in order to improve the situation.

Tickets

Signify that a human needs to take action, but not immediately. The system cannot automatically handle the situation, but if a human takes action in a few days, no damage will result.

Logging

No one needs to look at this information, but it is recorded for diagnostic or forensic purposes. The expectation is that no one reads logs unless something else prompts them to do so.

Emergency Response

Reliability is a function of mean time to failure (MTTF) and mean time to repair (MTTR). The most relevant metric in evaluating the effectiveness of emergency response is how quickly the response team can bring the system back to health—that is, the MTTR.
Humans add latency. Even if a given system experiences more actual failures, a system that can avoid emergencies that require human intervention will have higher availability than a system that requires hands-on intervention. When humans are necessary, we have found that thinking through and recording the best practices ahead of time in a "playbook" produces roughly a 3x improvement in MTTR as compared to the strategy of "winging it." The hero jack-of-all-trades on-call engineer does work, but the practiced on-call engineer armed with a playbook works much better. While no playbook, no matter how comprehensive it may be, is a substitute for smart engineers able to think on the fly, clear and thorough troubleshooting steps and tips are valuable when responding to a high-stakes or time-sensitive page. Thus, Google SRE relies on on-call playbooks, in addition to exercises such as the "Wheel of Misfortune,"7 to prepare engineers to react to on-call events.
TODO!

Change Management

70% of outages are due to changes in a live system. Best practices in this domain use automation to accomplish the following:
- Progressive roll-outs
- Quickly and accurately detecting problems
- Rolling back changes safely when problems arise