Managing real-time feed subscriptions from non-persistent streams—such as WebSockets, gRPC streams, or real-time pub-sub platforms—presents unique challenges in scalability and load balancing, especially when handling dynamic subscriber workloads.
In this talk, you'll learn how to harness Kafka consumer groups, rebalancing, and Kafka Streams lifecycle hooks to dynamically distribute and scale real-time subscription workloads.
We’ll explore effective strategies for managing subscription state, whether in internal Kafka state stores or external databases. By the end, you’ll have clear patterns and actionable insights for building resilient, dynamically scalable solutions to seamlessly bridge WebSocket data into Kafka.