in-depth session where we tackle the challenge of dynamically invoking Azure Data Factory (ADF) pipelines using Apache Airflow. While ADF does not natively support this functionality, Apache Airflow provides a powerful solution to orchestrate these pipelines effectively.

What You'll Learn:
Running ADF Pipelines from Airflow:

Setup and Configuration: We will guide you through the prerequisites, including necessary tools and accounts (Azure, ADF, Airflow), and provide a step-by-step configuration process for integrating Airflow with ADF.
Creating a DAG (Directed Acyclic Graph): Learn how to create a DAG in Airflow and configure tasks to invoke ADF pipelines.
Execution and Monitoring: Experience a live demonstration of executing the DAG, monitoring, and managing the pipeline execution.
Running Custom Modules in Airflow:

Module Development: Discover how to create custom modules tailored for specific tasks.
Integration with Airflow: Understand the process of adding custom modules to Airflow DAGs.
Execution and Monitoring: See a live demo of executing custom modules and observing the execution process.

Session Highlights:
Overcome the limitation of ADF's native capabilities by leveraging Apache Airflow.
Gain hands-on experience through two detailed demos.
Enhance your data orchestration skills with practical, real-world applications.

This session is perfect for data engineers, developers, and IT professionals looking to expand their knowledge and capabilities in data pipeline orchestration using cutting-edge tools.