Scaling SLI/SLO - Pushing Your Observability Platform To Its Limits

By now we are all very familiar with what an SLO is, what an SLI is, how one computes a burn rate and setting up alerts on top of them. But what does it really take to scale those when you move from a hundred to a thousand to tens of thousands of those evaluations? This talk focuses on problems we saw with running rules at scale and how we took the Prometheus rules manager and scaled it to be able to evaluate hundreds of thousands of rules every minute.