Session

Storm to Flink: Migrating Latency Sensitive Real-Time Streaming Applications with Zero Downtime

Migrating latency-sensitive real-time streaming pipelines that process millions of requests a day with zero downtime is challenging. Additional system properties such as no data loss, high availability, resilience, etc., make the migration process even more challenging. At Bloomberg, we have streaming pipelines (Compliance Pipelines) to surveil communication traffic in real-time, implemented using Apache Storm. In this talk, we will explore why we decided to move our Compliance Pipelines away from Storm, and why Apache Flink was chosen for this task. We will then go into depth about how we redesigned the application to leverage the framework's strengths. We will also discuss issues we encountered during the migration and how we overcame them. We will look at different metrics that illustrate the performance improvements of the updated application. Finally, we will wrap up with a discussion related to testing, productionizing the application, and going live.

Niharika Sakuru

Software Engineer at Bloomberg LP

New York City, New York, United States

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top