Modern data platforms promise scalability and flexibility—but without careful optimisation, costs can spiral and performance can degrade. This session focuses on how to effectively optimise your Azure Databricks Lakehouse to deliver faster data pipelines while significantly reducing platform costs.
The intent of this session is to provide a practical, experience-led view of how data engineers can design, build, and operate efficient lakehouse architectures. I will explore common inefficiencies across storage, compute, and pipeline design, and show how small changes in approach can unlock major gains in performance and cost control.
Key topics will include optimising Delta Lake storage patterns, improving Spark workload performance, right-sizing clusters, and making effective use of Databricks features such as Photon, caching, and job orchestration. I will also cover how to design pipelines with efficiency in mind from the outset, rather than retrofitting optimisations later.
In addition, I will discuss how governance and observability play a critical role in cost management, helping teams understand where spend is occurring and how to control it without slowing down innovation.
Attendees will leave with practical techniques, patterns, and best practices they can immediately apply to make their Databricks environments faster, more reliable, and more cost-efficient.