Ask The Experts: Cloud

Join us at the Speakers Booth in the Expo Hall to find your favorite speakers hanging out here during conference breaks and ask your questions. Take this opportunity to get the answers you are looking for!

Efficient Data Partitioning with Microsoft Fabric: Best Practices and Implementation Guide

the session contains an information about Microsoft Fabric SAAS service, i will explain about the main components such as oneLake,DF ,Synapse data engineering and much more.
after the brief introduction, I will explain about partitioning in data lakehouse, what is the difference between lakehouse and warehouse and how to organize data using medallion architecture pattern, this pattern is used by data engineers to logically store data in multiple layers to guarantee a clean and smooth transition till we have Gold data which represents the final refinement that data has to have before displaying in PowerBI / engaging to a third part such as data scientists for ML operations
A short demo is presented to the audience just to get a better understanding of how to implement the architecture in Microsoft Fabric service
and in the end we will have about 10 minutes for Q&A

Step-by-Step Guide: Building ETL Workflows in Azure cloud Services

building ETL's using Azue Data Lake , Azure Data Factory

Mastering Data Workflows: Microsoft Fabric's Metadata Pipeline Solution with the Medallion Architect

Microsoft Fabric provides both Data Lakehouse and Data Warehouse platforms for Data Analytics. In a separate post, I illustrated a Metadata Driven Pipeline pattern for Microsoft Fabric following the medallion architecture with Fabric Data Lakehouses used for both the Bronze and Gold layers and SQL views over tables for the Silver layer. Fabric Data Lakehouse is perfect for landing data into the Bronze layer. And if I can host my star schema in a Fabric Data Lakehouse, why consider Fabric Data Warehouse as the Gold layer? Because Fabric Data Warehouse has some features that may be a better fit for your organization.

Microsoft Fabric: All About The Data Analytics Platform

A breif introduction about fabric components and main concepts (OneLake, OneSecurity,V-order optimization and much more) and a short demo on how to execute ETL's using Fabric different components.

Step-by-Step Guide: Building ETL Workflows in Microsoft Fabric

In this session, we will delve into building robust ETL (Extract, Transform, Load) pipelines using Microsoft Fabric. This demonstration aims to enhance our data insights by leveraging powerful features of Microsoft Fabric.

Throughout the session, participants will explore the full capabilities of Microsoft Fabric, focusing on:

Workspace: Setting up collaborative environments for efficient data management.
Lakehouse and Warehouse: Constructing the Medallion architecture for unified data storage and processing.
Data Flow Gen2: Designing scalable and efficient data processing workflows.
Pipelines: Automating the flow of data from source to destination.
Notebooks: Using interactive notebooks for exploratory data analysis and pipeline development.
SQL Endpoints: Integrating structured query capabilities for advanced data manipulation.
Datasets: Organizing and managing data assets for streamlined access and analysis.
PowerBI Reports: Visualizing insights through interactive and insightful reports.
By the end of the session, attendees will have gained practical experience in harnessing these features to build resilient ETL pipelines, empowering them to extract maximum value from their data resources.

Prerequisites
Internet connectivity
Your own active Azure Subscribtion
Your own Microsoft Fabric Trial

Mastering the Art: Orchestrating ADF with the Power of Workflow Orchestration Manager

in-depth session where we tackle the challenge of dynamically invoking Azure Data Factory (ADF) pipelines using Apache Airflow. While ADF does not natively support this functionality, Apache Airflow provides a powerful solution to orchestrate these pipelines effectively.

What You'll Learn:
Running ADF Pipelines from Airflow:

Setup and Configuration: We will guide you through the prerequisites, including necessary tools and accounts (Azure, ADF, Airflow), and provide a step-by-step configuration process for integrating Airflow with ADF.
Creating a DAG (Directed Acyclic Graph): Learn how to create a DAG in Airflow and configure tasks to invoke ADF pipelines.
Execution and Monitoring: Experience a live demonstration of executing the DAG, monitoring, and managing the pipeline execution.
Running Custom Modules in Airflow:

Module Development: Discover how to create custom modules tailored for specific tasks.
Integration with Airflow: Understand the process of adding custom modules to Airflow DAGs.
Execution and Monitoring: See a live demo of executing custom modules and observing the execution process.

Session Highlights:
Overcome the limitation of ADF's native capabilities by leveraging Apache Airflow.
Gain hands-on experience through two detailed demos.
Enhance your data orchestration skills with practical, real-world applications.

This session is perfect for data engineers, developers, and IT professionals looking to expand their knowledge and capabilities in data pipeline orchestration using cutting-edge tools.