Danish Rehman
Attentive - Staff Software Engineer
San Bruno, California, United States
Actions
Danish is a Staff Software Engineer at Attentive, where he plays a key role in shaping the architecture and scalability of our product. With nearly 15 years of experience in building distributed systems for high traffic ad-tech and e-commerce, he has been instrumental in developing our next-generation marketing solutions powered by AI. Outside of work, Danish enjoys gardening, vermiculture, and rock climbing. He loves sharing his harvest with his four-year old son before they head out on climbing adventures together.
Area of Expertise
Topics
Deprecating Distributed Locks for Low Latency Event Consumption
This session covers how Attentive scaled mutual exclusivity in its event-driven architecture by evolving from Redis-based locks to Apache Pulsar’s FAILOVER and KEY_SHARED subscription modes. Operating at the scale of 620M texts/day during peak events, we share how we addressed the “split-brain” effect, improved the latency of event consumption, and navigated the trade-offs across subscription models. We’ll also dive into key lessons learned in building a reliable, low-latency, high-throughput message orchestration system.
Goodbye Distributed Locks: Message Orchestration at Scale with Apache Pulsar
Summary of the talk:
At Attentive, we’ve built a large-scale, event-driven messaging platform powered by Apache Pulsar to support the dynamic demands of user shopping behavior. Last year, during peak events like Black Friday / Cyber Monday, our platform scaled to handle 3.9 billion sms and emails, including 620 million texts on Black Friday alone.
In this talk, we’ll walk through how Attentive evolved its high-throughput, event-driven architecture to achieve mutual exclusivity at scale using Apache Pulsar. Specifically, we’ll cover:
- How the need for user-level mutual exclusivity became critical for preserving orchestration integrity and delivering a consistent user experience.
- Challenges with Pulsar’s SHARED mode, great for scaling but caused mutual exclusivity violations and inconsistent decisions due to concurrent event consumption ("split-brain effect").
- Our architectural evolution to achieve mutual exclusivity - moving from Redis-based distributed locks to Pulsar’s FAILOVER and eventually KEY_SHARED subscription mode.
- The trade-offs, operational hurdles, and real-world production incidents that shaped our understanding of Pulsar’s subscription modes and their impact on throughput and fault tolerance.
Key Takeaways
- How Apache Pulsar can be leveraged to achieve mutual exclusivity in event-driven architectures
- Trade-offs between Redis-based distributed locks and Pulsar’s subscription modes (SHARED, FAILOVER, KEY_SHARED), including the pros and cons of each approach for ensuring mutual exclusivity and scalability in event-driven systems
- Strategies to mitigate the above trade-offs.
Note: Based on our blog post https://tech.attentive.com/articles/goodbye-distributed-locks
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top