Paul McComish
Databricks Chamption and Principal Data Engineer
Poole, United Kingdom
Actions
Paul McComish is a Principal Data Engineering Consultant at Advancing Analytics based in the UK, with extensive experience working across data engineering, analytics, and modern cloud platforms. He has spent many years helping organisations design and deliver data solutions, from traditional analytics environments through to modern lakehouse architectures.
Paul holds multiple Microsoft and Databricks certifications, is a Databricks Champion and is actively engaged in the data community, regularly sharing insights and perspectives on topics such as data platform modernisation, migration from legacy systems, and evolving data engineering practices.
Area of Expertise
Topics
Cut Costs, Boost Speed: Optimising Your Azure Databricks Lakehouse
Modern data platforms promise scalability and flexibility—but without careful optimisation, costs can spiral and performance can degrade. This session focuses on how to effectively optimise your Azure Databricks Lakehouse to deliver faster data pipelines while significantly reducing platform costs.
The intent of this session is to provide a practical, experience-led view of how data engineers can design, build, and operate efficient lakehouse architectures. I will explore common inefficiencies across storage, compute, and pipeline design, and show how small changes in approach can unlock major gains in performance and cost control.
Key topics will include optimising Delta Lake storage patterns, improving Spark workload performance, right-sizing clusters, and making effective use of Databricks features such as Photon, caching, and job orchestration. I will also cover how to design pipelines with efficiency in mind from the outset, rather than retrofitting optimisations later.
In addition, I will discuss how governance and observability play a critical role in cost management, helping teams understand where spend is occurring and how to control it without slowing down innovation.
Attendees will leave with practical techniques, patterns, and best practices they can immediately apply to make their Databricks environments faster, more reliable, and more cost-efficient.
Cut Costs, Boost Speed: Optimising Your Azure Databricks Lakehouse
Modern data platforms promise scalability and flexibility—but without careful optimisation, costs can spiral and performance can degrade. This session focuses on how to effectively optimise your Azure Databricks Lakehouse to deliver faster data pipelines while significantly reducing platform costs.
The intent of this session is to provide a practical, experience-led view of how data engineers can design, build, and operate efficient lakehouse architectures. I will explore common inefficiencies across storage, compute, and pipeline design, and show how small changes in approach can unlock major gains in performance and cost control.
Key topics will include optimising Delta Lake storage patterns, improving Spark workload performance, right-sizing clusters, and making effective use of Databricks features such as Photon, caching, and job orchestration. I will also cover how to design pipelines with efficiency in mind from the outset, rather than retrofitting optimisations later.
In addition, I will discuss how governance and observability play a critical role in cost management, helping teams understand where spend is occurring and how to control it without slowing down innovation.
Attendees will leave with practical techniques, patterns, and best practices they can immediately apply to make their Databricks environments faster, more reliable, and more cost-efficient.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top