In the fast-evolving landscape of real-time data processing, choosing the right stream processing framework is paramount. Two years ago, we embarked on an ambitious journey to migrate our entire data processing stack to Apache Flink, driven by the promise of ultimate flexibility, open-source community support, and granular control over our streaming infrastructure.

This talk chronicles our experience of operating a large-scale Flink deployment, managing dozens of complex pipelines processing terabytes of data daily. We will take a technical deep dive into the challenges we encountered at scale, including the operational overhead of managing Flink clusters on Kubernetes, the complexities of state management and checkpointing for massive stateful applications, and the nuances of performance tuning in a multi-tenant environment.

However, as our data ecosystem matured, the very control that drew us to Flink became a significant operational burden. This session will candidly explore the critical factors that led to our strategic decision to migrate back to a managed service, specifically Google Dataflow. We will present a comparative analysis of the two frameworks, focusing on total cost of ownership (TCO), developer productivity, and operational resilience.

Attendees will gain valuable insights into the trade-offs between self-managed and serverless stream processing. We'll share our migration playbook, covering everything from initial pipeline compatibility assessments to the execution of a zero-downtime cutover. This is a story of pragmatism over purity, offering practical lessons for any organization navigating the complex decision of which stream processing engine will truly allow them to sail, not just scale.