Shivam Yadav
SDE-2 ShareChat
Actions
Shivam is an accomplished Software Development Engineer at ShareChat, currently working with the Livestream Data Science team. Shivam owns and maintains the backend services that deliver personalized livestream and chatroom feeds, ranked in real-time to millions of users.
Previously, Shivam worked on ShareChat’s Central Data Platform, where he led the development and maintenance of a Kafka Streams-based counters framework, enabling real-time aggregation and updates of organizational metrics.
With extensive experience in Apache Kafka, GCP Pub/Sub, and Redpanda, Shivam has a deep understanding of real-time data processing and distributed systems, and is passionate about building scalable and efficient streaming solutions.
In today’s session, Shivam will share his expertise and insights into leveraging Kafka for real-time personalization at scale.
Unleashing Data Powerhouses - Benthos and Kafka Streams
In this session, we'll unravel the story of ShareChat's transition from Java to Benthos for crafting efficient ETL pipelines. With a single configuration file, Benthos effortlessly connects diverse sources and sinks, transforming the way we handle data. Discover how, armed with "at least once" guarantees, Benthos emerged as the go-to solution for our stateless pipelines.
Focused on real-world use cases, we'll explore how Benthos, paired seamlessly with Kafka and Kafka Streams, became the linchpin of our operations. From sending notifications to triggering events for millions of users, to routing posts for reviews, Benthos proved its mettle in simplifying complex tasks.
Learn how we platformized Benthos at ShareChat, deploying over 20 jobs from a single repository. Delve into the specifics of how Benthos with Kafka and Kafka Streams powers our data pipelines, handling tasks such as dumping data to databases, making API calls, and routing events to different message queues. The session will highlight the simplicity and performance benefits we've achieved—processing 2K events/sec to an impressive 45K events/sec—without the need for extensive developer code.
Streaming Aggregations at Scale using Kafka and Kafka Streams
Windowed aggregation on streams is a powerful concept for analytical data processing, especially in cases where waiting hours or even minutes for data to be available is inconceivable. While most people think of aggregations as an analytical requirement, they also help trim down data to size and can be critical to scaling systems without leading to ballooning costs. We had a similar use case in ShareChat where we had hundreds of thousands of counter increments (updates)/sec for everything from the number of views on a post to revenue numbers. Our databases could not keep up with the write volume, and there were frequent hot-spotting issues. Furthermore, the data was often inconsistent due to multiple writers, and taking locks further added to our misery.
To solve this issue, we used Kafka and Kafka Streams to build an aggregation framework that can handle hundreds of thousands of counter increments per second. Streaming aggregation helped us batch updates and helped reduce the write throughput in our DBs. Further, it helped solve our hot-spotting issues and eliminated the need to take locks.
This talk discusses the challenges of building the platform, managing an in-house Kafka setup, and the lessons learned from tuning Kafka Streams. Furthermore, we discuss how we optimised the solution to scale to even 1M updates/sec with zero hiccups or manual involvement.
Today, the offering forms an integral part of our core streaming platform, and the talk will be helpful for developers who have similar requirements for streaming aggregations or want to know more about event-driven architectures using Kafka and Kafka Streams.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top