Shipeng Xie
Software Engineer
Palo Alto, California, United States
Actions
Software Engineer, focusing on data engineering and infra with Flink, k8s micro services development.
Links
Area of Expertise
Managed Cloud Threat Detection with Flink
Cloud Detection enables the MITRE based threat detection for simple cases like for a single event with rule matching or complex scenarios like event correlation across multiple devices. It allows faster rules creation and publishing with real time matching using flink as a streaming system. Doing this in the cloud removes the cumbersome on premise component update, which saves the time for threat analysts and reduces costs for customers.
With Change Data Capture (CDC) and Outbox pattern, we reliably broadcast the threat detection rules from the rule microservice to each parallelism of the Flink job to match threat detection rules against events to promote them to alerts.
We also used Flink’s Complex Event Processing (CEP) to correlate multiple events by detecting certain patterns within a specific time window and generate alerts against detected complex threats corresponding to MITRE tactics.
Flink Alert Processing to Address Alert Fatigue in Cybersecurity
One of the biggest reasons that people leave security vendors is due to an overwhelming amount of alerts that Security Operations Center (SOC) analysts must triage. We use a couple of jobs managed by the Flink Kubernetes operator to help alleviate this issue by automatically reducing the amount of false positive alerts that SOC analysts need to handle.
Alert Classification is our real-time ML processing job that classifies alerts as either noisy or anomalous. We utilize both prevalence and clustering-based algorithms to generate a single final score for each alert via async operators that query external features and model inference endpoints.
Alert suppression is our other real-time alert processing job which maintains the human-defined rules in broadcast state and suppresses alerts. Consisting of multiple microservices and data stores, we used the Change Data Capture (CDC) pattern to reliably propagate the rules from the SQL database to the Flink engine via broadcast state.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top