Jeff Xiang
Senior Software Engineer at Pinterest
New York City, New York, United States
Actions
Jeff is a software engineer on the Logging Platform team at Pinterest. Over the years, he has built and contributed to many of the systems and projects that power Pinterest's data ingestion technology, including Kafka operation automation, data lineage visibility and monitoring, reliability improvements of cross-region data mirroring, and most recently building unified PubSub client.
In his spare time, Jeff enjoys reading, DJing, eating sushi, and hiking.
Area of Expertise
Topics
Streamlining Innovation: Service Abstraction and its Impact on Productivity and Cost Efficiency
In today's rapidly evolving landscape of data streaming innovations, organizations face both opportunities and challenges. At Pinterest, we have successfully navigated this terrain through the strategic application of service abstraction, achieving significant advancements in productivity and cost efficiency. This session will present our journey and innovations, demonstrating how we simplified releases and minimized disruption for hundreds of customer teams and applications.
Our centralized PubSub platform is maintained by a knowledgeable core team, supports thousands of pipelines, and offers uniform service standards. However, releasing new features transparently remains a challenge due to factors like software bugs, potentially diverting engineers from innovation to maintenance. To counter this, we pursued isolation of the PubSub layer, integrating cutting-edge products that make our platform more stable, reliable, and easy to onboard.
Key innovations include:
* MemQ, a remote storage PubSub system that offers up-to 90% reduction in infrastructure costs.
* Kafka Tiered Storage, that achieves over a 90% reduction in cost per GB stored.
* Enabling FlinkSQL for real-time, ad-hoc queries on streaming datasets to unify streaming and batch processes.
* PubSub Client, a unified client library that facilitates the above mentioned abstraction.
These open-sourced advancements enable customers to seamlessly embrace new technologies with ease and transparency. Our projects—MemQ, Pinterest Tiered Storage, and the PubSub Client—address common industry pain points, such as high switching costs, lack of integrations, and customer confusion. Attendees will gain valuable insights into how service abstraction can maximize adoption, minimize costs, and enhance customer satisfaction in large-scale data environments.
Pinterest’s Journey to a Automated, Efficient, and Low-Maintenance PubSub Platform
At Pinterest, Apache® Kafka runs on ~2000 brokers across 50+ clusters, moving data at petabyte scale every day. Through investing in automation and focusing on cost efficiency, we have seen positive results in dev velocity, scalability, stability, and cost footprint, enabling everyone to quickly and easily launch data streaming use cases that power business growth, while keeping our platform low-maintenance.
We believe that sharing the innovations along our journey to a highly-automated, efficient, and low-maintenance PubSub platform can provide immense value for industry partners. This talk will cover the following topics in-depth, bringing inspiration and actionable takeaways for each attendee who might face the same problems operating PubSub platforms at scale:
1. Automation: >80% reduction in overall onboarding time for PubSub applications
* Automated endpoint discovery with PubSub Client (PSC) via 100% completion of Java migrations to PSC, available standalone and as Flink connector (open-sourced)
* Automated resource provisioning for PubSub-based applications via drag-and-drop UI, reducing setup time from 30 mins to 5 mins
* Automated Kafka broker & cluster management and recovery via Orion (open-sourced)
* Automated client-side error handling via PSC (open-sourced)
2. Efficiency: >25% reduction in platform cost
* Reducing PubSub infrastructure cost via migrating large pipelines from Kafka to MemQ (open-sourced)
* Reducing Kafka cluster sizes and optimizing efficiency via using heterogeneous hardware
* Reducing Kafka broker footprint via storage tiering
* Reducing cross-region KafkaMirror resource utilization via Shallow Mirror (KIP-712)
3. Low-Maintenance: simplifying business processes
* Generic CDC framework via Debezium / Kafka Connect
* Improving cost visibility and attribution for multi-tenant environment, enabling cost-driven decision making
* Moving towards service tiering to further simplify customer onboarding and platform operations
Pinterest’s Journey to a Automated, Efficient, and Low-Maintenance Kafka Platform
At Pinterest, Apache® Kafka runs on ~2000 brokers across 50+ clusters, moving data at petabyte scale every day. Through investing in automation and focusing on cost efficiency, we have seen positive results in dev velocity, scalability, stability, and cost footprint, enabling everyone to quickly and easily launch Kafka-supported use cases that power business growth, while keeping our platform low-maintenance.
We believe that sharing the innovations along our journey to a highly-automated, efficient, and low-maintenance Kafka platform can provide immense value for industry partners. This talk will cover the following topics in-depth, bringing inspiration and actionable takeaways for each attendee who might face the same problems operating Kafka at scale:
1. Automation: >80% reduction in overall onboarding time for Kafka-supported applications
* Automated endpoint discovery with PubSub Client (PSC), available standalone and as Flink connector (open-sourced)
* Automated resource provisioning for Kafka-based applications via drag-and-drop UI, reducing setup time from 30 mins to 5 mins
* Automated broker & cluster management and recovery via Orion (open-sourced)
* Automated client-side error handling via PSC (open-sourced)
2. Efficiency: >25% reduction in platform cost
* Reducing cluster sizes and optimizing efficiency via using heterogeneous hardware
* Reducing broker footprint via storage tiering
* Reducing cross-region KafkaMirror resource utilization via Shallow Mirror (KIP-712)
3. Low-Maintenance: simplifying business processes
* Generic CDC framework via Debezium / Kafka Connect
* Improving cost visibility and attribution for multi-tenant environment, enabling cost-driven decision making
* Moving towards service tiering to further simplify customer onboarding and platform operations
Current 2024 Sessionize Event
Kafka Summit London 2024 Sessionize Event
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top