Back to videos

Automate based on frequency not recency

Beware of recency bias when automating incidents!
2 min
Summary

The most recent incident isn't a great predictor of the next thing that might happen. So when deciding what to automate, you must get over the urge to automate the most recent incident and focus on the frequent ones.

It's because those are the ones that:

  • reduce the most toil, giving you more bang for the buck.
  • you understand best because you fix them again and again.

Of course, it's a bit more complicated than what I just described. Sometimes you want to automate something because:

  • you don't want operators to access customer data on their machines.
  • you want to reduce the chance of someone making a consequential mistake with a manual change.

But mostly, it's bang for the buck. When I was at AWS, I'd make my service teams automate 1 incident per week - just 1. But if you do that consistently, you'll be down about 60% over the year, maybe more. That matters a great deal!

Part of what we did with Shoreline is reducing the cost of the cost-benefit equation so that you don't have to think that hard about what to automate next. SREs are busy, so automation won't happen unless it's easy to do. That's why most automations in Shoreline can be built in a few hours. Fixing things forever only takes about 2x the time as fixing it one time. I really want it to take the same or less time.

But it's already an order of magnitude better than the other systems I know. You just need to be disciplined about continuing this process of automation: One incident every on-call rotation. That's how you improve availability and reduce toil for yourself, your colleagues, and your customers.

Transcript

View more Shoreline videos

Looking for more? View our most recent videos
2 min
What I Learned at AWS About Managing SLOs
All we could do was apply automation to fix the issues when they occurred, providing customers much better availability.
13 min
Shoreline Incident Automation Demo
See Shoreline in action, debugging an incident and automating remediations in a fraction of the usual time.
3 min
How to boost reliability without hiring more SREs
How can companies increase reliability without hiring an army of engineers?