Flink Alert Processing to Address Alert Fatigue in Cybersecurity

One of the biggest reasons that people leave security vendors is due to an overwhelming amount of alerts that Security Operations Center (SOC) analysts must triage. We use a couple of jobs managed by the Flink Kubernetes operator to help alleviate this issue by automatically reducing the amount of false positive alerts that SOC analysts need to handle.

Alert Classification is our real-time ML processing job that classifies alerts as either noisy or anomalous. We utilize both prevalence and clustering-based algorithms to generate a single final score for each alert via async operators that query external features and model inference endpoints.

Alert suppression is our other real-time alert processing job which maintains the human-defined rules in broadcast state and suppresses alerts. Consisting of multiple microservices and data stores, we used the Change Data Capture (CDC) pattern to reliably propagate the rules from the SQL database to the Flink engine via broadcast state.