Most data engineering projects are larger then a single fact table and one or two dimensions. Probably more then a handful sources. While most samples out there are built around how to get a source file into a table the reality is that, in real life data engineering is much more then that.

But having tens or hundreds of sources that all must be copied into a staging table often means repetition. So it will not be long before most people resort to a meta data driven framework or just roll their own. And it is not new, I saw projects build this way 20 years ago. Simply because it gets the job done.

This session will show what the downside might be when using meta data driven pipelines and what possible solutions you have to work around these. Hopefully you will walk away with some new insight you can apply to your data pipelines right away.