Apoorv Mittal
Apache Kafka Committer | Staff Engineer @ Confluent
London, United Kingdom
Actions
Apoorv Mittal is a Apache Kafka Committer and Staff Software Engineer at Confluent with over 15 years of experience in the tech industry. He has been primarily involved in developing distributed systems and architecting e-commerce platforms. He has worked extensively in companies which have emerged as a unicorn where Apoorv joined in initial growth phase and helped developing core platforms to support the business.
Links
Topics
Taking Queues for Kafka into production
Apache Kafka is more than just the de facto event streaming platform. It's becoming the universal platform for data in motion. But there’s one aspect which we wanted to make easier.
Perhaps you have struggled with how best to consume data from Kafka when your application just wants a bunch of messages rather than a stream of events. If you're using consumer groups today, you'll know that you need to worry about balancing the number of partitions and consumers. And what’s the best way to cope with individual bad records? These are the reasons which motivated KIP-932: Queues for Kafka.
KIP-932 introduces the concept of share groups. Share groups let your applications consume data from regular Kafka topics with fine-grained sharing and per-message acknowledgement, just like a traditional queue. Now you can bring your queuing workloads to Apache Kafka without doing all the hard work yourself.
KIP-932 is in preview in Apache Kafka 4.1 but it's fast progressing towards production-readiness. Come and learn how to build applications using share groups, how to monitor share groups using the broker and client metrics, and how to tune them so you're ready to take your first workloads into production.
Queues for Kafka
Event streaming is great but sometimes it’s easier to use a queue, especially when parallel consumption is more important than ordering. Wouldn't it be great if you had the option of consuming your data in Apache Kafka just like a message queue?
For workloads where each message is an independent work item, you’d really like to be able to run as many consumers as you need, cooperating to handle the load, and to acknowledge messages one at a time as the work is completed. You might even want to be able to retry specific messages. This is much easier to achieve using a queue rather than a topic with a consumer group.
KIP-932 brings queuing semantics to Apache Kafka. It introduces the concept of share groups. Share groups let your applications consume data off regular Kafka topics with per-message acknowledgement and without worrying about balancing the number of partitions and consumers. With this KIP, you can bring your queuing workloads to Apache Kafka.
Come and hear about this innovative new feature being added to Apache Kafka 4.0.
Know the metrics to monitor your Kafka clients
The multitude of metrics generated by Kafka clients can overwhelm developers, making monitoring and troubleshooting a challenging task. Varying workloads across multiple clients create a complex landscape of metrics, demanding careful prioritization from developers to ensure effective monitoring.
Also learn how to explore the power of KIP-714 for efficiently exporting client metrics and visualizing them in your preferred dashboard.
Come and demystify the basics of Kafka clients (Producer and Consumer) metrics. Gain mastery and take control of your data pipeline.
Centralized monitoring and troubleshooting for Kafka clients
Kafka clients collect a lot of metrics that can be used for troubleshooting. When you have ten clients in a cluster, it's already hard to coordinate collection of the metrics. When you have thousands of clients, it's really hard. Many cluster operators do not have control over the clients, and the lack of consistent telemetry across clients is an operational gap.
KIP-714 improves monitoring and troubleshooting of client performance by enabling clients to push metrics to brokers. The cluster operator can define subscriptions for the metrics they want to collect and from which clients, and then the clients push the metrics. The KIP also standardizes metrics so that metrics from different client implementations can easily be aggregated together.
Come and hear how this KIP makes it easy to identify problematic clients, and watch a live demo showing how you can use it with your Apache Kafka cluster.
Community Over Code NA 2025 Sessionize Event
Current 2024 Sessionize Event
Apoorv Mittal
Apache Kafka Committer | Staff Engineer @ Confluent
London, United Kingdom
Links
Actions
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top