In today's rapidly evolving landscape of data streaming innovations, organizations face both opportunities and challenges. At Pinterest, we have successfully navigated this terrain through the strategic application of service abstraction, achieving significant advancements in productivity and cost efficiency. This session will present our journey and innovations, demonstrating how we simplified releases and minimized disruption for hundreds of customer teams and applications.

Our centralized PubSub platform is maintained by a knowledgeable core team, supports thousands of pipelines, and offers uniform service standards. However, releasing new features transparently remains a challenge due to factors like software bugs, potentially diverting engineers from innovation to maintenance. To counter this, we pursued isolation of the PubSub layer, integrating cutting-edge products that make our platform more stable, reliable, and easy to onboard.

Key innovations include:
* MemQ, a remote storage PubSub system that offers up-to 90% reduction in infrastructure costs.
* Kafka Tiered Storage, that achieves over a 90% reduction in cost per GB stored.
* Enabling FlinkSQL for real-time, ad-hoc queries on streaming datasets to unify streaming and batch processes.
* PubSub Client, a unified client library that facilitates the above mentioned abstraction.

These open-sourced advancements enable customers to seamlessly embrace new technologies with ease and transparency. Our projects—MemQ, Pinterest Tiered Storage, and the PubSub Client—address common industry pain points, such as high switching costs, lack of integrations, and customer confusion. Attendees will gain valuable insights into how service abstraction can maximize adoption, minimize costs, and enhance customer satisfaction in large-scale data environments.