Anat Stolarsky
Building bridges between data and insight!
Tel Aviv, Israel
Actions
Anat Stolarsky is a highly experienced Data Engineer at Next Insurance, where she leads a team of Data Engineers focused on marketing, partnership, and growth data.
She has over 10 years of experience in multiple sectors, including AdTech, Finances, and Cloud Computing. She loves solving complex data challenges and implementing advanced technology solutions.
Anat lived in Switzerland for 5 years during which she enjoyed hiking in the Swiss Alps and learning German.
Area of Expertise
Topics
From Backyard Sports to Olympic Champs - Data Interface Architecture for Data-Driven Organizations
In our fast-moving engineering world, where data evolves quickly, organizations must constantly process data from multiple sources of different domains at scale. Data Engineering teams must develop and maintain business-ready, easy-to-use, and robust data pipelines that continuously support multiple data consumers with their data needs.
How can development teams move fast while not breaking existing pipelines or logic? How can we provide self-service options to consumers without an ongoing dependency on Data Engineering? Who should be responsible for defining business data logic and models? Is it data producers, data consumers, or data engineers?
To address these challenges and pain points at Next Insurance, we started an ambitious project called “Data Interface”. In our talk, we will present the architectural principles and design decisions we took. We will describe how we define robust and stable data contracts and how we incorporated the bronze/silver/gold design pattern in our data lake architecture to support the different needs of various data consumers. We will share our success stories and insights from this innovative approach for rolling out a data interface architecture.
Eliminating blinds spots in Airflow - Quality score analysis for workflows
A key requirement for data pipelines is that they produce high quality data.
At Next Insurance we use quality gate tasks as circuit breakers within our workflows, but we were struggling to measure their effectiveness. There's no "effectiveness" indication in Airflow, so in order to evaluate our workflow's quality, we built a graph analysis tool to analyze its quality gates.
In this talk, we will show how our tool drives quality by calculating a score for each workflow, giving us a bottom line metric for tracking improvements, analyzing the flow, highlighting gaps in coverage and providing statistics on gate effectiveness.
I'll cover the methodology behind the tool as well as the pitfalls and edge cases, and show how we integrated the tool into our CI/CD systems.
By applying these concepts to your own processes, you too will be able to improve visibility into your workflow's quality and set KPIs for future improvement.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top