One of the biggest challenges with deploying new Flink pipelines is to come up with a suitable Flink configuration for the underlying Flink cluster. In the process, users are faced with an overwhelming amount of configuration options, most of them non-trivial to configure unless they deeply understand Flink internals.

Not surprisingly, most users defer to a trial-and-error approach for configuration options, a process which is both time-consuming and probably non-optimal.

We present Flink Autotuning, a novel approach to automatically generating an efficient Flink configuration and dynamically adjusting it over time to fit the needs of the workloads.

In its initial implementation, Flink Autotuning focuses on the Flink memory configuration, which users report as the most frustrating aspect of the configuration process. The complexity arises from the various memory pools that Flink utilizes. Incorrect configuration of any of these pools can lead to application crashes or inefficient memory usage.

In this talk, we introduce the ideas and concepts behind Flink Autotuning. We describe how we implemented Flink Autotuning and how we integrated it with existing solutions like Flink Autoscaling. During a demo part, we will see it working in action. Finally, we’ll talk about future improvements.