Back to videos

Automate Based on Frequency not Recency

Beware of recency bias when automating incidents!
2 min

The most recent incident isn't a great predictor of the next thing that might happen. So when deciding what to automate, you must get over the urge to automate the most recent incident and focus on the frequent ones.

It's because those are the ones that:

  • reduce the most toil, giving you more bang for the buck.
  • you understand best because you fix them again and again.

Of course, it's a bit more complicated than what I just described. Sometimes you want to automate something because:

  • you don't want operators to access customer data on their machines.
  • you want to reduce the chance of someone making a consequential mistake with a manual change.

But mostly, it's bang for the buck. When I was at AWS, I'd make my service teams automate 1 incident per week - just 1. But if you do that consistently, you'll be down about 60% over the year, maybe more. That matters a great deal!

Part of what we did with Shoreline is reducing the cost of the cost-benefit equation so that you don't have to think that hard about what to automate next. SREs are busy, so automation won't happen unless it's easy to do. That's why most automations in Shoreline can be built in a few hours. Fixing things forever only takes about 2x the time as fixing it one time. I really want it to take the same or less time.

But it's already an order of magnitude better than the other systems I know. You just need to be disciplined about continuing this process of automation: One incident every on-call rotation. That's how you improve availability and reduce toil for yourself, your colleagues, and your customers.


View more Shoreline videos

Looking for more? View our most recent videos
3 min
Actively Managing Systems to Improve Utilization
We're all being asked to do more with less now a days. For those of us in production operations, one of the best ways we can do that is eliminate waste with automation to drive higher utilization.
14 min
theCUBE Interviews Shoreline CEO Anurag Gupta at AWS re:Invent
Anurag Gupta joined John Walls to discuss innovation in the cloud with DevOps teams for the Global Startup Program at AWS re:Invent 2022.
2 min
The Best Way to Improve Your On-Call
No one wants to do on-call because you can't control when the incident happens. Improve your on-call by building automations that eliminate common production incidents.