Apache Flink has become the go-to stream processing engine for large-scale real-time analytics, but operating it at petabyte scale introduces unique challenges. How can you ensure that your Flink SQL jobs are fast, resource-efficient, and cost-effective?

In this talk, we will dive into the core principles of how we optimised Flink SQL queries for a European Bank to 4x performance using simple techniques. Drawing insights from real-world workloads and best practices, we will explore:
• Understanding Flink SQL Execution: How Flink translates SQL queries into execution plans and why performance bottlenecks occur.
• Optimising Joins and Aggregations: Strategies for handling stateful operations efficiently, minimising data shuffling, and leveraging Flink’s internal optimisation techniques.
• Memory and State Tuning: How to fine-tune Flink’s state backend, memory management, and parallelism to prevent excessive checkpointing overhead.
• Practical Techniques for Performance Tuning: Real-world debugging workflows using Flink’s Web UI, metrics, and profiling tools.
• Lessons from Scaling to Petabytes: How leading companies optimize Flink SQL in production, ensuring reliability and cost efficiency.

This session is aimed at data engineers, Flink practitioners, and architects looking to push the boundaries of Flink SQL performance. By the end of the talk, attendees will walk away with actionable strategies to reduce job latency, optimize resource usage, and scale Flink SQL jobs efficiently in production.

Whether you’re operating Flink for real-time analytics, ETL pipelines, or streaming machine learning, this talk will help you write faster, more efficient Flink SQL queries at scale.