AIOps: Finding that Needle in a Haystack using data science

We cannot manage today’s dynamic, constantly changing IT environments with yesterday’s tools. As our systems are getting modernized, they are generating more data than ever before. It’s easy to assume that more data is better. But at a certain point, the size and unstructured nature of data can make extraction of patterns and insights an insurmountable task for humans. In today’s competitive world, you cannot wait for clients to complain and address the issue reactively. You need to be proactive and have systems in place which can proactively identify and address the client's issues.

Cerner has 600+ client production domains, we collect 10TB+ of data per day just from the stability and performance monitoring tools for Millennium. Identifying anomalies from this data manually is almost impossible. To solve this challenge, the performance & stability team has built a system called AIOps, which is driven by machine learning and statistical models. This system detects anomalies in data in real-time to flag potential client issues and alert all the stakeholders along with potential resolution and the root cause of the issue to quickly address the issue. This system has detected more than 2500+ client incidents in real-time along with potential root causes, which has helped us reduce client escalations significantly and reduce our mean time to resolution by 35%. A lot of client stakeholders have appreciated the value generated from these alerts.

In this talk we would like to cover:

1. The workflow of the AIOps system along with deep-dive of major use cases
- How we have built anomaly detection models to flag potential issues
proactively using Local Outlier Factors, Isolation Forests, Gaussian Mixture
Model, and other statistical approaches
- How we have implemented similar software issues prediction model in AIOps
to avoid duplicate investigations which are helping us save 4FTEs per year by
reducing the duplicate investigations

2. Walkthrough of Anomaly Detection Driven Actions and client success stories
3. How other teams can AIOps services to improve their operations