PoAn Yang
ASF, Apache Kafka / YuniKorn committer
Actions
I am an open-source software enthusiast, primarily focusing on Apache Kafka. I am among the top 20 contributors, and my work involves enhancing the AsyncKafkaConsumer and developing the next-generation group coordinator.
Links
State of Apache YuniKorn
Apache YuniKorn is a lightweight, universal resource scheduler for running batch workloads in Kubernetes. YuniKorn extends the scheduling capabilities of Kubernetes and supports a wide variety of workloads such as batch processing, data analytics, and machine learning tasks, with robust support for frameworks such as Spark, Flink, and TensorFlow.
YuniKorn 1.0.0 was released in 2022 and contained many major features, e.g. App-aware scheduling, Hierarchy Resource Queues, Gang Scheduling. Since then, the project has continued to grow, and until the latest release, 1.5.0 (released on March 14, 2024), many new features have been included, such as Preemption, User Quota Enforcement, and an Event System.
In this talk, we will provide a basic introduction to the project and focus on the changes and improvements that have occurred in recent releases. Besides this, we also want to look into the future and highlight some topics the community is currently working on and give an insight into the roadmap for planned features.
标题：Apache YuniKorn 的现状
Apache YuniKorn 是一个轻量级的、通用的资源调度器，用于在 Kubernetes 中运行批处理工作负载。YuniKorn 扩展了 Kubernetes 的调度能力，并支持各种工作负载，如批处理、数据分析和机器学习任务，同时对诸如 Spark、Flink 和 TensorFlow 等框架提供了强大的支持。
YuniKorn 1.0.0 于 2022 年发布，包含了许多重要功能，例如应用感知调度、层次化资源队列和队列调度。在最新的 1.5.0 版本（于 2024 年 3 月 14 日发布），新增了许多特性，如抢占、用户配额执行和事件系统。
在本次演讲中，我们将对该项目进行基本介绍，并重点讲述近期版本中的变化和改进。此外，我们还将展望未来，重点介绍社区目前正在关注的一些主题，并对计划特性的路线图提供一些见解。
Kafka 4.0 Highlights: Features, Removals, and Upgrades
While there’s already a lot of buzz around Kafka 4.0, most discussions focus on the headline features—like the removal of ZooKeeper or the introduction of new APIs. In this session, we want to highlight some of the lesser-known but equally important changes in Kafka 4.0. These updates might not grab the spotlight, but they can significantly impact how you configure, operate, and debug Kafka in real-world environments.
One of the major questions users face is how to approach the upgrade to Kafka 4.0. KIP-896 removes support for protocol versions 2.0 and below, and KIP-1124 offers official guidance for upgrading Kafka clients, Connect, and Streams. On the server side, users must ensure JDK compatibility and consider how to transition from ZooKeeper-based setups to the new architecture. This section will cover the most critical considerations to keep in mind during the upgrade process.
The Kafka test framework has also undergone a major evolution. Previously, tests were written using an inheritance-heavy design, where subclasses inherited cluster setup logic from parent classes. This made tests difficult to understand and debug, often introducing hidden dependencies and unnecessary configuration. The new test framework simplifies this by using JUnit annotations like @ClusterTest, making test logic clearer, more modular, and easier to maintain.
Kafka 4.1 will also introduces smarter behavior in how it handles rack-aware rebalancing. In earlier versions, storing rack information consumed a large portion of memory for group metadata—up to 80% in some cases. KIP-1101 introduces a caching mechanism that detects rack topology changes efficiently, reducing memory usage and avoiding unnecessary recalculations during group rebalances.
Finally, Kafka 4.0 completes a long-awaited logging upgrade with the delivery of KIP-653. Originally proposed in 2020, KIP-653 replaces reload4j with log4j2, finally resolving long-standing CVE concerns and modernizing the logging infrastructure. Alongside this, Kafka drops support for .properties log config files in favor of YAML. This section provides an overview of the migration and what users need to watch out for when adopting log4j2.
Community Over Code Asia 2025 Sessionize Event
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top