The Lakehouse and The Medallion Architecture

This session is focused on Data Engineering and the evolution of the different architecture paradigms over the course of the last decades.
We’ll talk about:
• The evolution of the Data Warehouse to Data Lake and to Data Lakehouse.
• The Medallion Architecture.
• The most important Data Engineering principles to adopt in your Data Platform, e.g.:
• Data Validation
• Data Pipeline Dependencies
• Referential Integrity

Microsoft Fabric Real-Time Intelligence for Data Engineers

In the world of dynamically changing data, the key challenge is to efficiently and automatically process information in real-time. During this session, I will show how to integrate Lakehouse and real-time processing architectures on the Microsoft Fabric platform, using functionalities such as Fabric Eventstream and Activator. I'll present a practical use case in which data loading and processing automation allows for a quick overview and supports the creation of an analytical platform. You will learn how to help businesses to build scalable and efficient real-time solutions.

Azure DevOps from the field

In the past Azure DevOps (or VSTS actually) was not that great tool when it comes to managing projects, collaboration,
not mentioning area of deployment. In result many people had to complement it with tools like TeamCity, Octopus, etc.
This changed.
For many years, Azure DevOps is very mature, versatile and enough for most Azure projects.
During this workshop, I want to share my practical experience from the field, from practical projects across several customers I worked with.
We will go through the portal from A to Z, learning purpose of each module: Overview, dashboard, Wiki, Boards (features, user stories, tasks, sprints).
I will explain how to start working in organisation, what's the scope of organisation, project, Teams and (what's even more important) how to organise code in GIT repository, when building data platform (as an example), splitting code and pipelines into Infra and App pieces.
I will touch topics like ARM Template, BICEP, Terraform, explain CI/CD and show how to prepare build and release pipelines.
We will see the most popular tasks and approaches to achieve a goal with minimal efforts having some templates and working examples.

Azure Data Factory - Deployment challenges

ADF is an important brick in the architecture of any modern data warehousing solution and many other scenarios.
As it exists for some time now and we know its capability pretty well, the deployment of the service is still something that leaves much to be desired, specifically in a bit more complex instances.
In this session, I will show a few challenges to publishing ADF and solution for them.

ADF Deployments with Azure DevOps

Azure Data Factory is a great orchestration tool in the cloud, is mature and for a while now with us.
Authoring the pipelines and other objects as a developer via browser (v2), working appropriately with branches, debug mode and understanding an integration with Git repo might be a bit tricky.
If you add to this, the need for deployment to different environments, adf_publish branch and why actually two methods of deployment exist - these things can be overwhelming.
Learn the best ways of working with ADF, scripts and tools for deployment and differences between them. See, how to automatically (not via UI) generate/export arm template files and use them in further steps in Azure DevOps, if you prefer using this way.

Delta Lake Tables 101

There are more and more file formats nowadays: Parquet format is not the best shiny star any longer. Now, the Delta Lake takes the prim.
Why people do confuse it with Parquet and always talk about files in this case?
In this session, we'll take a look at the evolution of ETL into ELT and its storage aspect, which explain why it is "a must" for modern data warehouse solutions and how is it related Delta Lake technology in cloud environments like Databricks or Synapse Analytics. Finally, we'll check what's Delta-Parquet creature presented in Microsoft Fabric OneLake recently.
We will see also what data layers (stages) are commonly set up and why they make sense.

In this session, we'll take a look at the evolution of ETL into ELT and its storage aspect, which explain why it is "a must" for modern data warehouse solutions and how is it related Delta Lake technology in cloud environments like Databricks, Synapse Analytics or the newest Microsoft Fabric.

SDLC for databases as a part of nowadays DevOps (pre-conf)

Nowadays the DevOps is the topic number one in many industries and companies. However, in some case, you see that the code repository is or will be the most important topic. The number of tasks could be overwhelming at first glance, however, there is no other way - you have to use the new tools and solutions
The code repository term is not really new but the way it is integrated with the database world is still sometimes questioned. And there are a lot more than that just naming Continuous integration, Continuous Delivery or Continuous Deployment.
I would like to show you tools to efficiently manage database projects (SQL Server Data Tools), how to start working with projects, how to deal with problems you will probably see during daily operations and how to configure and manage the projects' deployment process on different environments.

We will go to the Software Development Life Cycle (SDLC) process in great details from the database point of view. But we will not spend too much time on analysis but rather on the development part. I would like to show the usage of the Azure DevOps application.
Of course, there will be an entire module about best practices and how to efficiently use them in the database projects.
In the end, I would like to touch the cloud and show how to migrate the existing on-premises database to the Microsoft Azure SQL Database and how no to get into troubles.
After attending the workshop you will be able to do & know:
* create empty or import existing databases
* resolve various problems during the import of database
* manage database project and its objects
* handling CLR objects
* store data in a project (static data, master, etc.)
* what should be a part of a project and what shouldn't (Linked Servers, security)
* where and how to keep SQL Jobs
* split database project into more chunks and why it's required sometimes
* cope with unlimited amount of projects
* Avoid known issues like: temp tables, triggers, circular references, OPENQUERY, lack of validation
* Migrate project to Microsoft Azure (Cloud!)
* Use a hybrid approach
* Apply tSQLt Unit Tests
* Make a deployment manually and automatically (Azure DevOps)
* Distinguish (finally) all three types of "Continuous"
* some helpful PowerShell scripts

We are going to go through commercial tools as well as some tips & tricks.

Prerequisites:
Basic knowledge about: TSQL, Visual Studio & Azure DevOps;

The workshop will be done in the Visual Studio 2022 with the newest SSDT installed but you can use older version of the Visual Studio as well.
You can take your laptop with you as some of the task I'm going to do with you!
You will have access to all codes and slide deck.

Each participant of the workshop will get full, lifetime access to my "Database projects with SQL Server Data Tools (SSDT)" course worth $195 on my platform "Learn with SQLPlayer".

Fully automated CI/CD of SQL database within Azure DevOps

When working on a database in SSDT, there is a need to deploy our changes to further environments and at the same time maintain the consistency of databases between environments. During the session, I will present how we can publish the solution manually and then go to the Continuous Integration and Continuous Deployment process using the Azure DevOps environment (formerly VSTS). In addition, we will work on inserting the unit tests, approval steps and the others using Pester and PowerShell in order to gain full automation in our database deployment process.

SSDT allows you to import and maintenance a database project within Visual Studio. Prepare a few steps more to test and deploy changes and data into target SQL Server with Azure DevOps pipelines.

Microsoft Fabric Overview - where to start?

Microsoft Fabric is a unified software-as-a-service (SaaS) data platform built around a Data Lakehouse architecture.
But do you know how to use it, how to make it work for you?
In this session, I will walk you through various personas available in Fabric: Data Factory, Data Engineering, Data Science, Data Warehouse, Real-Time Analytics & Data Activator. Which one should you use? Let's check together what problems and architectures can we address with Microsoft Fabric and whether we can build a reliable Data Platform efficiently.