Speaker

Michael Robson

Michael Robson

Data Educator for Advancing Analytics | Newcastle DPAC UG Organiser | He/Him

Darlington, United Kingdom

A humble Data Guy with the best job in the world, I get to help people with a passion for data grow technically and professionally, thanks to the team at Advancing Analytics.
Previously a veteran of SQL Server turned Data Engineer, a proud member of the Data Community a regular helper at SQLBits, and formally one of the team behind Data Relay. More importantly, i'm a Husband, Dad of 2 people 1 dog and I love running #runhappy

Area of Expertise

  • Information & Communications Technology

Topics

  • Databases
  • Data Platform
  • Data Engineering
  • Database Administration
  • Azure Data & AI
  • Azure Data Lake
  • Data Analytics
  • Microsoft Data Platform
  • Azure SQL Database
  • Big Data
  • All things data

Exploring and Understanding Databricks Lakehouse Monitoring

Data lakes have become a core part of our data solutions. We push immense amounts of data, from various sources into our lakes on a daily, hourly, and even minute by minute, second by second basis. This data is constantly evolving, and must be nurtured and explored for us to understand it. Our applications consuming this data must be able to adapt as the data grows and evolves, in particular when we are working with Machine Learning Models.

To achieve this we must understand our data integrity, be able to visualise and alert on how the data changes over time, but what tooling can we use to achieve this? Databricks have recently announced a new feature that solves this problem: Databricks Lakehouse Monitoring. Databricks Lakehouse Monitoring lets you create time granular observations and set up custom metrics on your Data Lake. In particular, Databricks Lakehouse Monitoring allows you to analyse the statistical distribution of the data, and look for drift between the current data and a known baseline. It allows you to see when ML model inputs and predictions are shifting over time, and identify model performance trends.

This all sounds great, and a solution to our problem, but how good is Databricks Lakehouse Monitoring? What does and doesn’t it give us? How much will it cost? In this session we will explore Databricks Lakehouse Monitoring, digging into its key features, analysing how they can work for us, so we can better understand our data integrity.

By the end of this session you will have a better understanding of Databricks Lakehouse Monitoring, how to implement it, and if it is a good fit for you.

So, you want to be a Data Engineer Part 2 – Becoming a Data Engineer

In this session, we're going to get really specific about what you need to know to become a Data Engineer. Data Engineering is cool because you work with so much tech from Python to Data Lakes and Spark to name but a few, but mastering every technology completely is not essential. Together we'll look at the specific areas of the technology that you need to get started and also look at how to adapt the knowledge and expertise you already have to join us in the world of Data Engineering.

Databricks for the Middle Aisle

Databricks, if you didn't already know, is an industry-leading tool for processing massive quantities of data and is now one of the most in-demand skills for data professionals. But, what if you've never used it and you don't have a monthly budget on a cloud platform to spend each month, how do you get started?
One option is Databricks Community Edition, this session will show you how to get started with Community Edition from sign-up, loading data into the platform to producing some data insights and whatever else we can do in 50 minutes. Best of all everything that you learn should cost less than a middle aisle coffee maker.

Michael Robson

Data Educator for Advancing Analytics | Newcastle DPAC UG Organiser | He/Him

Darlington, United Kingdom

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top