Mastering Data Workflows: Microsoft Fabric's Metadata Pipeline Solution with the Medallion Architect

Microsoft Fabric provides both Data Lakehouse and Data Warehouse platforms for Data Analytics. In a separate post, I illustrated a Metadata Driven Pipeline pattern for Microsoft Fabric following the medallion architecture with Fabric Data Lakehouses used for both the Bronze and Gold layers and SQL views over tables for the Silver layer. Fabric Data Lakehouse is perfect for landing data into the Bronze layer. And if I can host my star schema in a Fabric Data Lakehouse, why consider Fabric Data Warehouse as the Gold layer? Because Fabric Data Warehouse has some features that may be a better fit for your organization.

Microsoft Fabric: All About The Data Analytics Platform

A breif introduction about fabric components and main concepts (OneLake, OneSecurity,V-order optimization and much more) and a short demo on how to execute ETL's using Fabric different components.

Step-by-Step Guide: Building ETL Workflows in Microsoft Fabric

In this session, we will delve into building robust ETL (Extract, Transform, Load) pipelines using Microsoft Fabric. This demonstration aims to enhance our data insights by leveraging powerful features of Microsoft Fabric.

Throughout the session, participants will explore the full capabilities of Microsoft Fabric, focusing on:

Workspace: Setting up collaborative environments for efficient data management.
Lakehouse and Warehouse: Constructing the Medallion architecture for unified data storage and processing.
Data Flow Gen2: Designing scalable and efficient data processing workflows.
Pipelines: Automating the flow of data from source to destination.
Notebooks: Using interactive notebooks for exploratory data analysis and pipeline development.
SQL Endpoints: Integrating structured query capabilities for advanced data manipulation.
Datasets: Organizing and managing data assets for streamlined access and analysis.
PowerBI Reports: Visualizing insights through interactive and insightful reports.
By the end of the session, attendees will have gained practical experience in harnessing these features to build resilient ETL pipelines, empowering them to extract maximum value from their data resources.

Prerequisites
Internet connectivity
Your own active Azure Subscribtion
Your own Microsoft Fabric Trial

Mastering the Art: Orchestrating ADF with the Power of Workflow Orchestration Manager

in-depth session where we tackle the challenge of dynamically invoking Azure Data Factory (ADF) pipelines using Apache Airflow. While ADF does not natively support this functionality, Apache Airflow provides a powerful solution to orchestrate these pipelines effectively.

What You'll Learn:
Running ADF Pipelines from Airflow:

Setup and Configuration: We will guide you through the prerequisites, including necessary tools and accounts (Azure, ADF, Airflow), and provide a step-by-step configuration process for integrating Airflow with ADF.
Creating a DAG (Directed Acyclic Graph): Learn how to create a DAG in Airflow and configure tasks to invoke ADF pipelines.
Execution and Monitoring: Experience a live demonstration of executing the DAG, monitoring, and managing the pipeline execution.
Running Custom Modules in Airflow:

Module Development: Discover how to create custom modules tailored for specific tasks.
Integration with Airflow: Understand the process of adding custom modules to Airflow DAGs.
Execution and Monitoring: See a live demo of executing custom modules and observing the execution process.

Session Highlights:
Overcome the limitation of ADF's native capabilities by leveraging Apache Airflow.
Gain hands-on experience through two detailed demos.
Enhance your data orchestration skills with practical, real-world applications.

This session is perfect for data engineers, developers, and IT professionals looking to expand their knowledge and capabilities in data pipeline orchestration using cutting-edge tools.

Key Takeaways from PySpark Notebooks in Microsoft Fabric

In this session, I will guide you step by step through the process of working with Notebooks and extracting data from APIs. Here's the approach we'll take:

Introduction to Notebooks and APIs: I will explain how Notebooks in Fabric provide a fast, efficient solution for data transformation and how APIs can serve as valuable data sources.

Setting Up the Environment: We'll start by installing and importing the essential libraries needed for working with PySpark and interacting with APIs.

Extracting Data from APIs: I'll demonstrate how to fetch data from an API, process it, and bring it into a DataFrame for further manipulation.

Creating a User Defined Function (UDF): I'll show you how to create a custom UDF in PySpark to handle more advanced data transformations. We will walk through the steps of defining the function, converting it into a UDF, and applying it to your DataFrame.

Complex Data Transformations: Finally, I will guide you on how to use UDFs to perform more complex transformations on your data, making it ready for other Fabric tasks.

Copilot: Understand how to use Copilot in notebooks for Data Engineering workloads to generate code snippets, provide explanation for existing code, suggest data visualizations, suggest analytical machine learning models, and more.

By the end of this session, you’ll have a clear understanding of how to leverage Notebooks and APIs in Fabric to perform complex data transformations with PySpark and UDFs.

This session is suitable for beginners, although basic knowledge of Azure Synapse Analytics or Microsoft Fabric is recommended. While the session is primarily aimed at data engineers, data architects and stakeholders are also welcome to attend.