While there’s already a lot of buzz around Kafka 4.0, most discussions focus on the headline features—like the removal of ZooKeeper or the introduction of new APIs. In this session, we want to highlight some of the lesser-known but equally important changes in Kafka 4.0. These updates might not grab the spotlight, but they can significantly impact how you configure, operate, and debug Kafka in real-world environments.

One of the major questions users face is how to approach the upgrade to Kafka 4.0. KIP-896 removes support for protocol versions 2.0 and below, and KIP-1124 offers official guidance for upgrading Kafka clients, Connect, and Streams. On the server side, users must ensure JDK compatibility and consider how to transition from ZooKeeper-based setups to the new architecture. This section will cover the most critical considerations to keep in mind during the upgrade process.

The Kafka test framework has also undergone a major evolution. Previously, tests were written using an inheritance-heavy design, where subclasses inherited cluster setup logic from parent classes. This made tests difficult to understand and debug, often introducing hidden dependencies and unnecessary configuration. The new test framework simplifies this by using JUnit annotations like @ClusterTest, making test logic clearer, more modular, and easier to maintain.

Kafka 4.1 will also introduces smarter behavior in how it handles rack-aware rebalancing. In earlier versions, storing rack information consumed a large portion of memory for group metadata—up to 80% in some cases. KIP-1101 introduces a caching mechanism that detects rack topology changes efficiently, reducing memory usage and avoiding unnecessary recalculations during group rebalances.

Finally, Kafka 4.0 completes a long-awaited logging upgrade with the delivery of KIP-653. Originally proposed in 2020, KIP-653 replaces reload4j with log4j2, finally resolving long-standing CVE concerns and modernizing the logging infrastructure. Alongside this, Kafka drops support for .properties log config files in favor of YAML. This section provides an overview of the migration and what users need to watch out for when adopting log4j2.