Do you have dozens or even hundreds of dataflows and tasks in your data pipelines in Microsoft Fabric? Especially if you've developed SSIS packages and now you're trying pipelines in Fabric, you will likely benefit from using a metadata-driven approach, which can save you significant ETL development effort and drastically reduce the complexity of your Fabric pipelines.

This session shows how to build pipelines in Fabric that are driven by metadata in JSON files in OneLake. Using these patterns, your ETL development velocity can be significantly increased.

Once you've built a pipeline to handle one table load, you can easily load dozens or hundreds of tables in your pipeline just by adding the metadata in OneLake.

I'll show you the patterns I use for managing dependencies (load these tables before those tables), incremental loading (load only the data that has changed since the last time this executed), and how to parameterize various options.

If you learn these patterns and implement them yourself, you can spend far less time coding ETL processes and increase your productivity greatly.