Summary of the talk:
At Attentive, we’ve built a large-scale, event-driven messaging platform powered by Apache Pulsar to support the dynamic demands of user shopping behavior. Last year, during peak events like Black Friday / Cyber Monday, our platform scaled to handle 3.9 billion sms and emails, including 620 million texts on Black Friday alone.

In this talk, we’ll walk through how Attentive evolved its high-throughput, event-driven architecture to achieve mutual exclusivity at scale using Apache Pulsar. Specifically, we’ll cover:

- How the need for user-level mutual exclusivity became critical for preserving orchestration integrity and delivering a consistent user experience.
- Challenges with Pulsar’s SHARED mode, great for scaling but caused mutual exclusivity violations and inconsistent decisions due to concurrent event consumption ("split-brain effect").
- Our architectural evolution to achieve mutual exclusivity - moving from Redis-based distributed locks to Pulsar’s FAILOVER and eventually KEY_SHARED subscription mode.
- The trade-offs, operational hurdles, and real-world production incidents that shaped our understanding of Pulsar’s subscription modes and their impact on throughput and fault tolerance.

Key Takeaways
- How Apache Pulsar can be leveraged to achieve mutual exclusivity in event-driven architectures
- Trade-offs between Redis-based distributed locks and Pulsar’s subscription modes (SHARED, FAILOVER, KEY_SHARED), including the pros and cons of each approach for ensuring mutual exclusivity and scalability in event-driven systems
- Strategies to mitigate the above trade-offs.

Note: Based on our blog post https://tech.attentive.com/articles/goodbye-distributed-locks