Sebastian Steinau
Software Architect/DevOps/Fabric
Tübingen, Germany
Actions
Dr. Sebastian Steinau is a software developer and architect specializing in APIs, distributed systems, and cloud-native solutions. He holds a PhD in Computer Science, where his research focused on developing distributed process management systems. With deep expertise in Microsoft Azure, DevOps practices, and Infrastructure as Code, he designs and implements scalable, resilient software architectures.
In recent years, he has expanded his focus into the data engineering space, particularly around Microsoft Fabric. He applies his DevOps and IaC expertise to data platforms, enabling version-controlled, automated, and reproducible data infrastructure deployments.
He is co-founder of SteinauTech Data & Software Solutions GmbH, a consultancy offering specialized services in data and software engineering. The company focuses on data projects for Microsoft Dynamics 365, combining technical excellence with practical business solutions.
Area of Expertise
Topics
Fabric, Git, & Pipelines - A Chord in Harmony
Nowadays with the rise of Microsoft Fabric a data engineer has the option to store data in warehouses, lakehouses, or KQL databases. As if we had to tell you that, that's the natural order of things. But there are other things a data engineer needs that do not belong in that kind of storage. Where do the SQL scripts, the notebooks, or the KQL query sets go?
If you don't know better, your local hard drive. If you are very bold, you might store them in Microsoft SharePoint. Or what's good enough for your data is good enough for your scripts, after all this saves on storage costs. Luckily, there is a better option: Git.
The scripts and notebooks are stored in a Git repository. Git is a distributed version control system initially developed for the Linux kernel. It allows for easily versioning your files and for collaboration on the same notebook. Git is a tool that has many amazing features, but like any tool it requires knowledge to use. Make your first steps into Git as a data engineer here and learn what Git can do for you when you develop a data solution. We show working with Git inside and outside of Microsoft Fabric with a focus on applicability and best practices.
Ok, now that your SQL scripts and your Fabric notebooks are fully versioned with Git, how to get them from the Git repository onto the Fabric workspaces? Of course, you could copy them manually, but wouldn't it be nicer if they just deployed automagically to the right place whenever a change occurs? The DevOps world has a solution: DevOps Pipelines. Learn how a pipeline can take your notebook and other Fabric items and deploy them to dev, staging, and prod environments, adapting connection strings and other parameters to match each environment automatically. We demonstrate with Microsoft Fabric how to automate your workflow so you can directly see the benefit.
Git and DevOps pipelines have helped software engineering immensely. And it might just do the same for data. Taking the load off your back by automating tasks in Microsoft Fabric can make your daily life easier. And working properly with version control gives you safety and recovery from error for your Fabric items. Let Git and Pipelines shine together to make your Fabric endeavor brighter
DevOps for the humble data engineer
A data engineer stores data in warehouses, lakehouses or even a simple database. As if we had to tell you that, that's the natural order of things. But there are other things a data engineer needs that do not belong in that kind of storage. Where do the SQL scripts, the workbooks, or the little python program go?
If you don't know better, your local hard drive. If you are very bold, you might store them in Microsoft SharePoint. Or what's good enough for your data is good enough for your scripts, after all this saves on storage costs. Luckily, there is a better option: Git.
The scripts and workbooks are stored in a Git repository. Git is a distributed version control system initially developed for the Linux kernel. It allows for easily versioning your files and for collaboration on the same workbook. Git is a tool that has many amazing features, but like any tool it requires knowledge to use. Make your first steps into Git as a data engineer here and learn what Git can do for you when you develop a data solution.
Ok, now that your stored procedure SQL script and your Databricks/Fabric/Synapse workbook is fully versioned with Git, how do get it from the Git repository onto the SQL database or the Databricks workspace? Of course you could copy it manually, but wouldn't it be nicer if it just deployed automagically whenever a change occurs? The DevOps world has a solution: Pipelines. Learn how a pipeline can take your workbook and deploy it to dev, staging, and prod environments, adapting connection string and other parameters to match each environment automatically. Pipelines are also a fully integrated part of Microsoft Fabric, we will use for a practical demonstration.
Git and DevOps pipelines have helped software engineering immensely. And it might just do the same for data. At the very least they might just ease your daily life. Isn't that possibility worth knowing more about Git and Pipelines?
Conflict Management – The Git Way
Git is a version control system that is on the verge of becoming ubiquitous in the IT world. Many data platforms and tools nowadays provide an integration for Git or strive to adapt their formats to make them compatible with Git for reaping the full benefits of contemporary version control. Recent examples of this are the ‘.tmdl’ file format for Power BI projects and the support for Git built into Microsoft Fabric. Git enables a safer, better, and more productive way of working. Yet one does need to know how to work with Git and deal with the challenges that come from using Git.
One challenge stems from using branches in Git to independently work on different features. At one point or another, these branches need to be merged back into the main solution, which can lead to merge conflicts when files have been altered in different ways. Being able to resolve these merge conflicts in a consistent and correct way is crucial. Microsoft Fabric promotes the use of branches, yet severely lacks the capabilities to optimally deal with merge conflicts in the Fabric UI. One frequent recommendation is to avoid merge conflicts altogether, which, in all honesty, won’t work. You will need to deal with them, which is not that hard if you know what you’re doing.
We will explorer what kind of merge conflicts typically occur when working with Fabric items. We will discuss when the means of Fabric of resolving a merge conflict can be used and when they better not be used. In this case, we show advanced techniques to solving merge conflicts using a combination of Fabric and native Git capabilities. And sometimes the only solution is to work entirely within Git. Whatever it may be, this session will give you the knowledge to be able to deal with any kind of merge conflicts that may arise, from the most basic issue to more complicated conflicts.
Being able to work with Git is as essential for Microsoft Fabric as working with Spark notebooks or knowing delta parquet. And you will become acquainted with merge conflicts sooner than later, so why not be prepared when the need arises?
DATA BASH '24 Sessionize Event
SQL Konferenz 2024 Sessionize Event
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top