With the rapid growth in Cloud, the amount of telemetry that needs to be processed has grown exponentially. This presentation delves into use of Apache Kafka and StarRock to ingest millions of network records per day, enabling security teams to visualize the network topology of different services and identify anomalies . We will delve into

* Kafka Architecture that supports this scale.
* StarRock partitioning strategies for high frequency reads.
* The challenges we encountered in production and the solutions we implemented to overcome them.