Session

Data Contracts as Code on Databricks

Data contracts are agreements between the people who produce data (upstream systems, pipelines, teams) and the people who consume it (analytics, machine learning, applications). Data contracts define what a dataset should look like, how fresh it should be, who can access it, and what quality it must meet.

The problem is that most data contracts in enterprises live in the wrong place: architecture blogs, Confluence pages, YAML files nobody reads after the first sprint. A markdown file that says "order_id must not be null" does not stop a null from being written. A Confluence page that says "this table is updated hourly" does not alert anyone when the pipeline fails.

This session focuses on the technical enforcement layer of a data contract. Using Azure Databricks Unity Catalog and Lakeflow Spark Declarative Pipelines, I will demonstrate how six of the data contract attributes: schema, quality, usage policies, security, SLAs, and data description can be defined once, enforced automatically, monitored continuously, and versioned alongside your pipeline code in Git.

At the end of this session, you will leave with a GitHub repository containing contract templates and a practical framework for turning your existing governance policies into objects that Databricks can actually enforce.
The session is for data engineers and platform teams working with Azure Databricks who want to bring their data contracts into the codebase.

Ceren Morey

Data Engineer

Brussels, Belgium

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top