This session covers a practical, end-to-end solution for managing and analyzing data in Azure. We’ll demonstrate how to build a lakehouse solution using Azure Databricks and Azure Machine Learning (ML) for data science. This presentation will contain loading and transformation strategies, structurizing of your data, and then prediction based on those data.

In the first part of this session, we’ll walk through setting up a data lakehouse in Azure Databricks, covering:

*A brief overview of the layers of the medallion architecture.
*Demonstrating how Slowly Changing Dimensions (SCD2) can be handled in Databricks notebooks.
*Organizing ETL workflows using Databricks Jobs.
*Setting up Unity Catalog and exploring the insights and data management capabilities it provides.

In the second part of this session, we’ll explore how to leverage the data prepared in Databricks for machine learning in Azure ML, covering:

*Integrating Azure Databricks data with Azure ML..
*Demo of how to implement prediction models in Azure ML.
*Managing the ML model lifecycle, from experimentation to deployment.
*Best practices for monitoring models after deployment.

This session is meant for data engineers and data scientists interested in these technologies. Attendees should have some In either setting of BI workloads or model development.