In our fast-moving engineering world, where data evolves quickly, organizations must constantly process data from multiple sources of different domains at scale. Data Engineering teams must develop and maintain business-ready, easy-to-use, and robust data pipelines that continuously support multiple data consumers with their data needs.

How can development teams move fast while not breaking existing pipelines or logic? How can we provide self-service options to consumers without an ongoing dependency on Data Engineering? Who should be responsible for defining business data logic and models? Is it data producers, data consumers, or data engineers?

To address these challenges and pain points at Next Insurance, we started an ambitious project called “Data Interface”. In our talk, we will present the architectural principles and design decisions we took. We will describe how we define robust and stable data contracts and how we incorporated the bronze/silver/gold design pattern in our data lake architecture to support the different needs of various data consumers. We will share our success stories and insights from this innovative approach for rolling out a data interface architecture.