Running large-scale streaming systems in production often reveals behaviors that never appear in small tests or synthetic benchmarks. In this talk we share lessons learned while operating high-throughput Apache Flink pipelines at scale, where subtle distribution mechanics created unexpected performance bottlenecks. Through real production incidents we will explore how data skew can emerge from different layers of the streaming stack: operator semantics, runtime key partitioning, and upstream source distribution. 

These experiences also raise a broader operational question: how do you continuously detect and investigate such issues in complex streaming systems?

In the second part of the talk, we explore a forward-looking approach to operating Flink at scale: embedding agents directly into the streaming environment to continuously analyze metrics, detect anomalies, and trigger automated investigations using LLMs and tool-based workflows. This enables the streaming platform itself to act as an intelligent observability layer, helping operators diagnose performance issues faster and reason about system behavior as workloads evolve.
Together, these perspectives combine practical production lessons with a glimpse into the future of AI-assisted operations for large-scale streaming systems.