Do you have dozens or even hundreds of datasets and tasks in your pipelines in Azure Data Factory (ADF)? Especially if you've developed SSIS packages and now you're trying Azure Data Factory, you will likely benefit from using a metadata-driven approach, which can save you significant ETL development effort and drastically reduce the complexity of your Azure Data Factories.

This session shows how to build pipelines in Azure Data Factory that are driven by data in a table. Using these patterns, your ETL development velocity can be significantly increased.

Once you've built a pipeline to handle one table load, you can easily load dozens or hundreds of tables in your pipeline just by updating the metadata in one table.

I'll show you the patterns I use for managing dependencies (load these tables before those tables), incremental loading (load only the data that has changed since the last time this executed), and how to parameterize various options.

If you learn these patterns and implement them yourself, you can spend far less time coding ETL processes and increase your productivity greatly.