Are you using Apache Kafka for real-time data processing? As your data grows, you'll need to scale up your Kafka clusters to keep up with the demand. One way to do this is by splitting a large Kafka cluster into smaller ones.

In this talk, I'll share some of my own experiences with splitting Kafka clusters, and explore two approaches in particular: splitting by domain and splitting based on the Cell architecture. Splitting by domain means dividing up your Kafka cluster by business area, while splitting based on the Cell architecture means dividing it up by identical clusters sharded by the ingress traffic. Both approaches have their own challenges, and I'll share some tips and tricks to help you avoid common pitfalls.

I'll examine key considerations, such as networking design, governance, and mirroring. The mirroring process, in particular, can become a bottleneck when scaling up, so I'll explore ways to optimize and automate mirroring to ensure optimal performance. I will also cover the governance aspect of scaling, including how to ensure consistency across multiple clusters and how to manage resources effectively.

To wrap things up, I'll share some real-world examples of successful Kafka cluster scaling, and the lessons learned from those experiences. By the end of this talk, you'll have a better understanding of how to scale up your Kafka clusters by splitting them, with practical insights and tips to help you achieve your goals.