As real-time data processing becomes crucial for modern enterprises, building resilient and fault-tolerant Apache Flink applications is more important than ever. This presentation explores the foundational and advanced strategies for ensuring high availability, reliability, and disaster recovery in Flink environments. 

We begin by demystifying Flink's checkpointing mechanism, its role in fast failure recovery, and common pitfalls to avoid. Beyond checkpointing, we delve into high-availability architectures, from active-passive to active-active setups, and their impact on system robustness. We also discuss the ongoing efforts of the Flink community towards blue-green deployment as a strategy to ensure seamless application updates with zero downtime. Finally, we present Bloomberg's solution for Kubernetes multi-cluster failover using Karmada.

Karmada is an open source project designed to simplify the deployment and management of cloud-native applications across multiple Kubernetes clusters. While it already provides mechanisms for workload failover, it has traditionally lacked support for preserving application state during disruptions, a critical requirement for stateful data processing workloads like Apache Flink.

To address this, Bloomberg’s Streaming Analytics Engineering team collaborated with the Karmada community to extend its capabilities for stateful recovery. This enhancement allows Karmada to not only failover workloads but also resume them with their state intact, making it an even more powerful solution for improving application resiliency across Kubernetes clusters.

By the end of this talk, attendees will gain actionable insights to build resilient, fault-tolerant Flink applications that can withstand real-world disruptions.

What product/technology does the talk focus on?
Apache Flink
Flink Kubernetes Operator
Karmada