Applying DevOps to Databricks can be a daunting task. In this talk this will be broken down into bite size chunks. Common DevOps subject areas will be covered, including CI/CD (Continuous Integration/Continuous Deployment), IAC (Infrastructure as Code) and Build Agents.

We will explore how to apply DevOps to Databricks, primarily using Azure DevOps tooling, but we will also explore the alternatives, including Github Actions. As a lot of Spark/Databricks users are Python users, we will look at the Databricks Rest API (using Python) to perform CI/CD tasks. For IAC we will primarily look at Terraform, but also explore other options, including ARM templates, Azure Bicep and Pulumi

As data professionals come with a variety of different backgrounds and skill sets, this talk will focus on providing options, and live demos, that demonstrate ways of achieving a DevOps solution that can be understood and maintained by everyone