Cost-effective resource assignment for a Flink deployment requires finding a configuration such that the deployment is neither under- nor overprovisioned. If done manually, this is a time-consuming and non-trivial exercise. By the time an optimal configuration has been found, it might already be suboptimal again.

Flink Autoscaling is a novel, research-backed open-source solution as part of the Flink Kubernetes Operator. Flink Autoscaling performs a deep introspection of the deployment to find a stable and cost-effective resource assignment. For each job vertex, it calculates the processing cost and capacity. These metrics are then used to scale each vertex based on the rate of incoming records and pre-existing backlog (e.g. pending records in Kafka). The result is a backpressure-free and cost-effective scaling decision, in a single pass.

Users who rolled out Flink Autoscaling to production pipelines reported a significant reduction in resource usage, maintenance and on-call burden.