Microsoft Fabric: Building a Data Ingestion and Processing framework to Drive Efficiency

Data pipelines are essential for moving and transforming data between different systems. However, managing a large number of data pipelines can be challenging and time-consuming. How can you ensure that your data pipelines are efficient, reliable, and consistent?
In this session, you will learn how to use a metadata-driven approach to manage your data pipelines and Notebook in Microsoft Fabric. Metadata is data about data, such as source, destination, schema and format.
By using metadata to define and control your data pipelines, you can achieve the following benefits:

1. Simplify and automate the creation and execution of data pipelines
2. Optimize the performance and scalability of data pipelines
3. Monitor and troubleshoot data pipelines

We will show you how to implement a Data Ingestion and Processing framework based on the Medallion Lakehouse architecture. We will also share the key learnings, best practices, and patterns that we have discovered from applying this framework in our own work.
All code used during the demo will be shared afterwards, so you can start building a framework directly after the session

Erwin de Kreuk

Data Platform MVP | Lead Data and AI |Public Speaker | InSpark | Innovate to Accelerate

Rotterdam, The Netherlands


Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top