Databricks & DevOps - How Declarative Automation Bundles help with CI/CD

Declarative Automation Bundles (DABs, previously known as Databricks Asset Bundles) bring an "infrastructure-as-code" approach to packaging and deploying notebooks, jobs, pipelines, and workflows across environments. Instead of clicking through the workspace UI or stitching together ad-hoc scripts, teams can define their entire Databricks project - code, configuration, and infrastructure - in YAML, version it alongside their source code, and deploy it consistently through a CI/CD pipeline.

In this session, we'll explore how DevOps practices translate into the Databricks world. We'll cover what Bundles are and how they compare to notebook-based deployments, how to structure a bundle for dev/test/prod environments, and how to wire them into a CI/CD pipeline (GitHub Actions/Azure DevOps) for automated validation and deployment. Along the way, we'll look at common pitfalls, testing strategies, and how Bundles fit into a broader data platform governance and release process.

You'll leave with a practical blueprint for automating Databricks deployments - reducing manual work, improving repeatability, and bringing real DevOps discipline to your data engineering workflows.

Grant, Revoke, Repeat: A Declarative Approach to Unity Catalog Permissions

Unity Catalog gives Databricks a proper permission model - but managing it by hand, click by click, doesn't scale. As the number of catalogs, schemas, and principals grows, so does the risk of drift, inconsistency, and permissions nobody can fully explain anymore.

This session starts with the fundamentals: how Unity Catalog's permission model works at the catalog and schema level, how privileges are inherited, and the principles worth designing around before you write a single line of automation. From there, we go practical: a demo of a custom, YAML-driven script that treats permissions as configuration - describing who should have access to what, then applying it as a set of GRANT and REVOKE statements against the target environment.

You'll see how a declarative, config-as-code approach turns permission management into something repeatable, auditable, and safe to run again and again - closing the gap between what your permissions should be and what they actually are.

Enforcing Data Quality at Every Layer: Lakeflow Declarative Pipelines Expectations in Databricks

Bad data doesn't announce itself - it slips quietly into dashboards, breaks downstream models, and erodes trust in your Lakehouse long before anyone notices the root cause. Lakeflow Declarative Pipelines (formerly Delta Live Tables) expectations let you catch it at the source, enforcing data quality rules directly inline as data moves through bronze, silver, and gold - without bolting on a separate framework.

This session is a hands-on look at LDP expectations: how to define them, the difference between warn, drop, and fail actions, how to quarantine and inspect violating records, and how to structure expectations across pipeline layers so quality gets stricter as data gets closer to production. We'll also look at how expectations surface in pipeline event logs and metrics, so failures are visible, not silent.

You'll leave knowing exactly how to design an expectations strategy for your own pipelines - catching bad data before it ships, without leaving the Databricks-native tooling you already have.

The Lakehouse and The Medallion Architecture

This session is focused on Data Engineering and the evolution of the different architecture paradigms over the course of the last decades.
We’ll talk about:
• The evolution of the Data Warehouse to Data Lake and to Data Lakehouse.
• The Medallion Architecture.
• The most important Data Engineering principles to adopt in your Data Platform, e.g.:
• Data Validation
• Data Pipeline Dependencies
• Referential Integrity

Azure Data Factory - Deployment challenges

ADF is an important brick in the architecture of any modern data warehousing solution and many other scenarios.
As it exists for some time now and we know its capability pretty well, the deployment of the service is still something that leaves much to be desired, specifically in a bit more complex instances.
In this session, I will show a few challenges to publishing ADF and solution for them.

ADF Deployments with Azure DevOps

Azure Data Factory is a great orchestration tool in the cloud, is mature and for a while now with us.
Authoring the pipelines and other objects as a developer via browser (v2), working appropriately with branches, debug mode and understanding an integration with Git repo might be a bit tricky.
If you add to this, the need for deployment to different environments, adf_publish branch and why actually two methods of deployment exist - these things can be overwhelming.
Learn the best ways of working with ADF, scripts and tools for deployment and differences between them. See, how to automatically (not via UI) generate/export arm template files and use them in further steps in Azure DevOps, if you prefer using this way.

Delta Lake Tables 101

There are more and more file formats nowadays: Parquet format is not the best shiny star any longer. Now, the Delta Lake takes the prim.
Why people do confuse it with Parquet and always talk about files in this case?
In this session, we'll take a look at the evolution of ETL into ELT and its storage aspect, which explain why it is "a must" for modern data warehouse solutions and how is it related Delta Lake technology in cloud environments like Databricks or Synapse Analytics. Finally, we'll check what's Delta-Parquet creature presented in Microsoft Fabric OneLake recently.
We will see also what data layers (stages) are commonly set up and why they make sense.

In this session, we'll take a look at the evolution of ETL into ELT and its storage aspect, which explain why it is "a must" for modern data warehouse solutions and how is it related Delta Lake technology in cloud environments like Databricks, Synapse Analytics or the newest Microsoft Fabric.

Microsoft Fabric Overview - where to start?

Microsoft Fabric is a unified software-as-a-service (SaaS) data platform built around a Data Lakehouse architecture.
But do you know how to use it, how to make it work for you?
In this session, I will walk you through various personas available in Fabric: Data Factory, Data Engineering, Data Science, Data Warehouse, Real-Time Analytics & Data Activator. Which one should you use? Let's check together what problems and architectures can we address with Microsoft Fabric and whether we can build a reliable Data Platform efficiently.

Fully automated CI/CD of SQL database within Azure DevOps

When working on a database in SSDT, there is a need to deploy our changes to further environments and at the same time maintain the consistency of databases between environments. During the session, I will present how we can publish the solution manually and then go to the Continuous Integration and Continuous Deployment process using the Azure DevOps environment (formerly VSTS). In addition, we will work on inserting the unit tests, approval steps and the others using Pester and PowerShell in order to gain full automation in our database deployment process.

SSDT allows you to import and maintenance a database project within Visual Studio. Prepare a few steps more to test and deploy changes and data into target SQL Server with Azure DevOps pipelines.