Maximilian Michels
Software Engineer
Berlin, Germany
Actions
Max is a software engineer at Apple who loves distributed systems and stream processing. Max previously worked on large-scale data processing tools and platforms at Google, Lyft, and Splunk. Max is a PMC member of Apache Flink and Apache Beam.
Links
Area of Expertise
Topics
Flink Autoscaling: A Year in Review - Performance, Challenges, and Innovations
At last year’s FlinkForward in Seattle, we presented Flink Autoscaling (FLIP-271), a research-backed open-source autoscaling solution as part of the Flink Kubernetes Operator. From day one, users who adopted Flink Autoscaling have reported a significant reduction in resource usage, maintenance, and on-call burden.
But it’s not all sunshine and rainbows. Since its inception, Flink Autoscaling has gone through a serious of improvements and additions. The originally proposed idea of performing a deep introspection of the Flink deployment to find a stable and cost-efficient resource assignment hasn’t changed, but the robustness of the algorithm and its metrics have been greatly improved.
One year later, is is time to evaluate Flink Autoscaling and present some of the challenges alongside with the latest additions and changes. In the course of this talk, we will show the inner workings of the Flink Autoscaling and highlight some of its unique integrations with Flink.
Flink Autotuning: Because Who Has Time to Manually Tweak Memory Settings?
One of the biggest challenges with deploying new Flink pipelines is to come up with a suitable Flink configuration for the underlying Flink cluster. In the process, users are faced with an overwhelming amount of configuration options, most of them non-trivial to configure unless they deeply understand Flink internals.
Not surprisingly, most users defer to a trial-and-error approach for configuration options, a process which is both time-consuming and probably non-optimal.
We present Flink Autotuning, a novel approach to automatically generating an efficient Flink configuration and dynamically adjusting it over time to fit the needs of the workloads.
In its initial implementation, Flink Autotuning focuses on the Flink memory configuration, which users report as the most frustrating aspect of the configuration process. The complexity arises from the various memory pools that Flink utilizes. Incorrect configuration of any of these pools can lead to application crashes or inefficient memory usage.
In this talk, we introduce the ideas and concepts behind Flink Autotuning. We describe how we implemented Flink Autotuning and how we integrated it with existing solutions like Flink Autoscaling. During a demo part, we will see it working in action. Finally, we’ll talk about future improvements.
From Pipeline To Execution - What happens when you run() your pipeline?
Apache Beam is a powerful framework: unified batch and stream processing, support for multiple execution engines, as well as writing code in multiple languages. It can be hard to wrap your head around how all of this works.
Fortunately, Beam's architecture can be broken down into several components which are easy to understand. Let's look at what happens when you run your Beam pipeline, how it gets translated, submitted, and executed. To make this more practical, we will use the Flink Runner as an example.
This talk is for people new to Beam who want to learn about Beam's architecture, as well as potential Runner authors eager to learn how to integrate with Beam.
Beam on Flink: How It Works and Why It Matters
Apache Beam is a data processing framework which conceptually is very similar to Flink. However, Beam does not have a full-blown runtime of its own. Instead, Beam programs run on execution engines such as Apache Flink, Apache Spark, or Google Cloud Dataflow.
Why is this interesting for Flink users?
Firstly, Apache Beam has multi-language support built-in. Want to write your data processing in Python or Go instead of Java? No problem with Beam.
Secondly, you write your Beam job only once with a batch/stream agnostic API. Then you are free to execute it on the execution engine of your choice. Besides Flink, do you have a Spark cluster that you want to run a batch job on, or do you want to try out Google Cloud Dataflow? Beam has you covered with no additional code changes.
You may ask: How does this work? How does a Beam program relate to a Flink program? What are the advantages and disadvantages of having the flexibility to choose the programming language and the execution engine?
In this talk, we will show how Beam and Flink work together, where they shine and where you might be better off with just using pure Flink.
Flink on Autopilot: How We Learned to Stop Worrying and Love the Autoscaling
Cost-effective resource assignment for a Flink deployment requires finding a configuration such that the deployment is neither under- nor overprovisioned. If done manually, this is a time-consuming and non-trivial exercise. By the time an optimal configuration has been found, it might already be suboptimal again.
Flink Autoscaling is a novel, research-backed open-source solution as part of the Flink Kubernetes Operator. Flink Autoscaling performs a deep introspection of the deployment to find a stable and cost-effective resource assignment. For each job vertex, it calculates the processing cost and capacity. These metrics are then used to scale each vertex based on the rate of incoming records and pre-existing backlog (e.g. pending records in Kafka). The result is a backpressure-free and cost-effective scaling decision, in a single pass.
Users who rolled out Flink Autoscaling to production pipelines reported a significant reduction in resource usage, maintenance and on-call burden.
Maximilian Michels
Software Engineer
Berlin, Germany
Links
Actions
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top