Back to videos

Automate based on frequency not recency

Beware of recency bias when automating incidents!
2 min
play_arrow
Summary

The most recent incident isn't a great predictor of the next thing that might happen. So when deciding what to automate, you must get over the urge to automate the most recent incident and focus on the frequent ones.

It's because those are the ones that:

  • reduce the most toil, giving you more bang for the buck.
  • you understand best because you fix them again and again.

Of course, it's a bit more complicated than what I just described. Sometimes you want to automate something because:

  • you don't want operators to access customer data on their machines.
  • you want to reduce the chance of someone making a consequential mistake with a manual change.

But mostly, it's bang for the buck. When I was at AWS, I'd make my service teams automate 1 incident per week - just 1. But if you do that consistently, you'll be down about 60% over the year, maybe more. That matters a great deal!

Part of what we did with Shoreline is reducing the cost of the cost-benefit equation so that you don't have to think that hard about what to automate next. SREs are busy, so automation won't happen unless it's easy to do. That's why most automations in Shoreline can be built in a few hours. Fixing things forever only takes about 2x the time as fixing it one time. I really want it to take the same or less time.

But it's already an order of magnitude better than the other systems I know. You just need to be disciplined about continuing this process of automation: One incident every on-call rotation. That's how you improve availability and reduce toil for yourself, your colleagues, and your customers.

Transcript

View more Shoreline videos

Looking for more? View our most recent videos
13 min
Shoreline Incident Automation Demo
See Shoreline in action, debugging an incident and automating remediations in a fraction of the usual time.
2 min
The Best Way to Improve Your On-Call
No one wants to do on-call because you can't control when the incident happens. Improve your on-call by building automations that eliminate common production incidents.
1 min
Shoreline End-to-end Automation
Easily and safely automate incident remediations with a few lines of code.