Selecting the best data transformation strategy

While the technologies used for transforming data have changed, the goals have not. Companies need data from multiple sources combined to provide the single version of the truth. What tools can be used to accomplish this? Microsoft Fabric contains a number of different options Azure data factory/Azure Synapse integration, Spark and Power BI Data flows can all be used to transform and model data. Which one should you use? This session will examine the different elements in your environment which can be used to determine which solution would be the best fit based upon elements such as data types, maintenance, cost, and skill levels.

Implementing Data Science in Microsoft Fabric

Developing data science solutions in Microsoft Fabric uses a number of components to create machine learning models and easily incorporate data from OneLake into dataframes. This session will also review some time saving tools added to Fabric including Data Wrangler and the Fabric Runtime. We will take a look at how you can incorporate these tools with notebooks to create a machine learning model and implement it in Microsoft Fabric data pipelines. In this session we will walk through the data process starting with data exploration, developing a dataset, experimenting and evaluating algorithms, modeling and implementing the solution.

Direct Lake and Direct Query In Power BI

Direct Query has been the choice for situations when businesses wanted large data models and where data entered in a database was immediately available. Direct Lake provides a new method for achieving the goals previously only available in Direct Query. Direct Lake was introduced to Power BI this year, and it can also be used to analyze large data volumes without needing to import the data, while providing good performance to users at scale who want to access the data. In this session we will see how you can use large data volumes in Power BI and take a look at how well it performs and scales using Direct Lake. We will examine how One Lake, lakehouses, and SQL Endpoints can be used together to provide optimal performance with Direct Lake. To see if this solution will work in your data environment, you will receive an understanding of the licensing required to implement this solution so that you can ensure that you have the knowledge you need.

Implementing Source control in Power BI

Creating a continuous integration and continuous delivery (CI/CD) in Power BI is possible now with PBIP files. In this session we will review how to implement it. We will walk through examples to provide attendees the information they need to add Power BI to CI/CD process used for other coding environments. See how you can automate assigning workspaces, adding users and managing deployments, and creating and editing pipelines.

3 ways to do Data Ingestion in Synapse and Fabric

In this session we will review 3 different methods for ingesting data, using the copy command, using Fabric's Dataflow2 and traditional Dataflows and discuss the reasons for using each method. The copy command has been updated and Dataflow2 is part of Fabric. We will review how these compare to traditional Synapse Ingestion dataflows and when it makes sense to use each.

Building an Fabric Environment for Power BI

Microsoft Fabric contains a number of different tools which you can use to create a data lake house environment perfect for Power BI. Learn the steps that are involved to develop and monitor data transformation pipelines, create a data lake house environment, and produce a final data store. Depending on the different data elements within the organization, the final data store could be a lakehouse, SQL Endpoint, data warehouse or Power BI model. We will review the different use cases so that users will understand which combination is best given properties of data and skillsets in your environment.

Incorporating Fabric One Lake in your data environment

With the introduction of Fabric One Lake in your data environment there is a lot of confusion regarding how it works. How do you make it work with GDPR if you need to ensure that data is not moved to another country? How do you implement the organizational structure when workspaces create folders within the lake? What kind of security can you employ with sensitive data to ensure that access is limited with One Lake? What kind of management needs to occur to the Parquet files which are automatically created? How do you use Power BI with One Lake? What is Direct Lake.This session provide answers to those questions and show you how Fabric One lake works with current and new resources.

Implementing Data Analytics with Microsoft Fabric

Microsoft Fabric has incorporated a number of different elements into one environment for data lakes with OneLake, data warehousing, machine learning, data transformation, and reporting. In this session we will look at using OneLake for data lakehouses and demonstrating how OneLake can be used instead of a database highlighting the performance improvements which have been made. Demonstrations will show the ability to use Fabric to connect to data from different sources and from different data lakes to ensure compliance with GDPR or other location-based regulations with OneLake. We will examine what a Fabric data lakehouse entails and how the data is integrated into Power BI for reporting. Changes in Spark clusters and Delta file implementation are examined to ensure you will understand how these improvements will impact your data movement pipelines and machine learning tasks. Data science workflows are reviewed to provide a good explanation of not only how to use them, but also the best practices for integrating these objects into pipelines. We will investigate how using the new Dataflow2 data transformations can speed up development and when they would be a good implementation choice. We will also examine the different ways Copilot is incorporated within Fabric so that your organization can be on the cutting edge of AI development with ChatGPT technology. This session will show you how to leverage Fabric’s different components for data driven decision making within your organization.

Data Science in Fabric

In this session we are going to look at creating a machine learning model and optimizing algorithm selection with Microsoft Fabric. After reviewing how that works, attendees will learn how they can either incorporate a low code solution to pick the best algorithm for the task. We will be examining the impact of Fabric Spark pools to those who may be familiar with Synapse or Databricks and how these changes improve the Spark development process. We will take a look at everything from creating to implementation incorporating new Fabric functionality.

Architecting a data solution in Fabric

With the introduction of Microsoft Fabric, the methods used for creating a data lake and using it in Power BI have changed. In this session we will take a look at how you can use the different components of Fabric to Architect a solution using the features of One lake with a Fabric workload. The session focuses on the differences with Fabric development to the methods you may have created a solution using Synapse and Power BI.

Working with Delta Lake in Azure Synapse

Spark Delta Lake moves the structure of a data lake closer to a database and is definitely something you are going to want to implement in Azure Synapse. This session will run through the code to employ Delta Lake throughout Synapse in Serverless, Spark and in Data flows. Working with Delta Lake in Azure Synapse can be tricky. The session will walk through how to create Spark tables in non-default locations and ensure the tables are added to your lake database. You will see how to use Delta Lake as a sink when Integrating data and review the options available. After watching you will understand how delta lake tables work when partitioned and how to work with Delta Lake in all of the environments in Azure Synapse.

Power BI Effective Management and Deployment Strategies Revealed

Project Management features in Power BI have changed quite a bit, and the new changes provide better opportunities for creating, managing and deploying solutions within your organization. Learn how to take advantage of the features in Power BI to improve collaboration, decrease model proliferation, and provide for a more robust set of deployment techniques.

Performance tuning Power BI reports

There are many reasons your Power BI reports may be running slowly. You may have a lot of data or inefficient DAX Calculations or a sub-optimal data model. Learn how to analyze the underlying issues with your reports and determine which solution you can use to improve the speed and scalability of your reports. We will examine solutions for import, direct query and live connections. No matter how you access your data, this session will provide you with some solutions that you can use on the reports in your environment.

Improving Power BI Performance with Data Modeling

Data modeling in Power BI is key to a successful implementation. In this session we will explore different techniques for improving the speed of Power BI by examining solutions to different common performance issues and look at how they can be improved with different modeling techniques. Attendees will learn how they can improve Power BI’s performance through the use of different table types, directional filtering and calculation groups. We will explore how different types of composite modeling can be used to improve the data environment and how these techniques can improve your Power BI apps.

How to pick the best Data transformation strategy

While the technologies used for transforming data have changed, the goals have not. Companies need data from multiple sources combined to provide the single version of the truth. What tools can be used to accomplish this? Azure data factory/Azure Synapse integration, SQL, Spark and Power BI Data flows can all be used to transform and model data. Which one should you use? This session will examine the different elements in your environment which can be used to determine which solution would be the best fit based upon elements such as data types, maintenance, cost, and skill levels.

How changes to Azure Synapse Analytics impact data architecture designs

The introduction of Azure Synapse Analytics has changed the architecture of many data solutions. Since the product was introduced, new features have been included which impact the use cases for when Azure Synapse would be a good choice in your data environment. We will review recent application modifications and how they influence what strategies and languages to use for orchestrating, storing and querying your data. This session may change your mind for when and how you can use Azure Synapse analytics as part of your data architecture.

Developing a Self-Service Power BI report development environment

Power BI is designed to be a reporting tool for the masses as report visualizations can be created with a few clicks. While it looks easy enough there several elements including the data model and DAX Measures which need to be created to make easy report creation possible. In this session we are going to explore the practices and methods which will make it possible for non-data professionals to develop their own reports in Power BI using models and templates created for that purpose. Attendees will learn what needs to be built to create a Power BI environment designed for end user report creation. Learning the elements which need to be included in the data model and what needs to be included in the report templates will provide a foundation for self-service development.

Data Model Design and Optimization for Power BI

If you have designed a data model for a data warehouse, you know most of what you need to develop a data model for Power BI, but there are some important differences to improve the overall performance and maintenance of Power BI. In this session we will review the differences and include design patterns for Row Level Security, data aggregations, and DAX Development. We will also review different design patterns for Composite Models, Direct Query, Import and Dual modes and how to organize them for optimal Power BI environments.

Moving away from One Model to rule them all

Often times a single data warehouse is meant to be the one solution to provide data to an organization, a method some may describe as the Lord of the Rings Model as there is one model to rule them all. There are a number of different reasons why this approach is not suited to most organizations. Some of the reasons are technical, others include an inability for one model to meet the needs of different groups of users or data types. In this session we will explore how to build a data environment which provides flexibility to different groups of users and examples of how to implement different types of technology to provide a broader set of solutions using Azure Synapse, Data lakes and Power BI.

Feedback Link - https://sqlb.it/?7013

Moving Away from Data Flows and Simplifying Data Pipelines in Azure Synapse Analytics

Data pipelines are commonly used to gather and transform data from different source systems. In Azure a common method for doing extract, transfer and load are data flows in Azure Data Factory or Integration Pipelines in Azure Synapse. While this method may be easy to code it is not the easiest thing to troubleshoot as there are many parts contained within it and it can be expensive to run. Conversely creating all of your ETL in Python can be difficult to write and maintain. In this session, you will learn how to use Spark SQL and Python to create notebooks which are called from integration pipelines to create an efficient, scalable, maintainable solution to create data migration and transfer tasks. You will want to incorporate this design pattern into your Azure data development environments when you see how easy it is to create and read this code.

Incorporating Data Lakes into Power BI

More and more companies are using data lakes as a central storage location for all things data. And because once you have data, people want to generate reports with it, this session will explore a number of different ways to integrate Azure Data Lake Gen 2 storage into a Power BI. We will review 3 different methods for integrating data from Azure Data Lake Gen 2 into Power BI. Learn how to incorporate data from Azure Data Lake storage with dataflows, using the Common Data Model, and directly accessing data into a Power BI model. In addition to seeing how to do this in the demos during this session, attendees will learn which method will be best suited for their environment.

Power BI DAX in a Day

If you have started working with DAX and would like to take your code to the next level, this session is for you. In this all day session, you will learn not only how to better write and understand DAX, but to use tools which can assist to write DAX faster and create better performing measures. This session will provide the ability to learn not only How but Why to use different DAX elements to be able to quickly develop complex measures.
The course will cover the following Topics
• Evaluation Contexts
• Improving DAX with Third-Party Tools and Best Practices
• Variable Uses and tricks
• Iterators
• Deep Dive into CALCULATE
• Advanced Time Intelligence
• Improving DAX Performance
Demonstrations and follow along exercises will provide the opportunity for you to not only hear about the concepts but see how they are implemented in code you can refer to later.

Decreasing development time with Azure Machine Learning in Azure Synapse Analytics

Algorithm selection and hyperparameter tuning can take a lot of time. There are many different options and testing all of them consumes a lot of resources. More and more people are looking to machine learning to help improve machine learning development. In this session, we will examine how to use machine learning to pick the best algorithm with Azure Synapse Analytics and Azure ML. The demonstrations will show how to use Apache Spark and Python in Azure Synapse together with Azure ML to create a machine learning solution. The solution will pick the best algorithm based on your criteria to determine what the important elements are. For example, you may be looking to create a regression experiment which has the best RMSE value and know there are some algorithms which you do not want to evaluate. You will see how to incorporate your criteria, tune your hyper parameters to select the best solution, and select the iterations and length of time you want the analysis to take. Once the algorithm is selected, the demonstrations will show how to incorporate the appropriate algorithm and deploy a solution which can be called in a development pipeline.

Data Lake Mangement with Azure Synapse and Delta Lake

Data lake management is required to ensure that the information stored can be readily analyzed. Spark Delta Lake moves the structure of a data lake closer to a database and is definitely something you are going to want to implement in Azure Synapse Analytics. In this session you will learn how to apply Spark Delta Lake to improve data quality, query speed, and review backups of files stored in the data lake. Implementing these strategies can improve analysis capabilities as data can be analyzed more like a data warehouse, without all of the transformation and storage costs. Using Delta Lake, attendees will see how to ensure the format is known when it was added, rather than finding out years later that no one is able to determine what is in the files. As data lakes can be queried like a database, we will examine how to speed analysis with Delta Lake by indexing flat files and consolidating the data to improve query performance. Need to look at what the data looked like prior to a change being made to the data in the lake? We will look at ways to travel back in time to review the data. This session will provide the skills needed to improve your data lake management with Delta Lake making it even easier to analyze data in Azure Synapse.

Utilizing Azure Synapse with Power BI

Learn how and why you can combine Power BI and Azure Synapse Analytics. Azure Synapse is Microsoft’s product for combining all sorts of data in the cloud and is designed to be the single source of data for the entire organization. As Azure Synapse’s install base increases it will become more and more important to learn how to use this important new data source with Power BI. In this session, we will look at configuring Synapse to incorporate Power BI workspaces, connecting and creating Azure Synapse data sources with Power BI, and data exploration with Power BI. The included demonstrations will show you how to get started with Azure Synapse and how you can perform these tasks in hour environment.

Using Azure Synapse Pools to Organize and Analyze Data

This session will provide a introduction into the different pools of Azure Synapse Analytics. We will look at how you can create a spark virtual database with Apache Spark Pools, create a data warehouse for large amounts of data with Dedicated Pools, perform ETL with Integrated Runtimes a type of Spark Pool, and create virtual databases and query flat files with Serverless Pools. We will look at how you can use these different pools to analyze and develop data in Azure.

Using Azure Synapse Analytics in your environment

Microsoft has introduced a new Azure data analytic product to better process and analyze large volumes of data, Azure Synapse Analytics. This session provides an overview to show the product in action. To better understand when you would want to use this product, the session includes scenarios and use cases to show where Azure Synapse Analytics would be the best tool for the job. There are two major components of the product, the newly renamed SQL DW and the Azure Synapse studio. Azure Synapse (formerly known as Azure SQL DW) now includes more data storage solutions, such as data lake and parquet file, and commands, like non-polybase COPY and PREDICT. We will explore these changes and others which you will need to know to provide better insight and performance when analyzing your data. Azure Synapse Studio incorporates a lot of other Azure features, including orchestration, spark, notebooks, and incorporating existing Machine Learning models. The session demonstrations will provide you a clear understanding of how to use these features in your environment by showing the steps needed to implement these components into a solution.

Sparking Azure Synapse for Machine Learning

Azure Synapse Workspace contains a number of tools under the one umbrella. In this session we are going to look at loading data into Data Lakes, exploring the data with on-Demand queries and create and use a Spark pool to analyze the data with machine learning. Attendees will see how easy it is to get started using Synapse if they haven't already and see how they can incorporate analyzing flat file data into their environments.

Teams Link: https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZDk0ZjE5YTEtMDg0ZS00ODYzLWI3YmItN2NhZDZkYThjY2Vh%40thread.v2/0?context=%7b%22Tid%22%3a%2272f988bf-86f1-41af-91ab-2d7cd011db47%22%2c%22Oid%22%3a%22c9f34171-87cd-4b3f-8458-3c6e9e43ffd9%22%7d

Introduction to Azure Synapse Analytics

Azure Synapse Analytics can be used to create and store a data warehouse, be used to query flat files, write ETL and write machine learning code with Spark. We we will review the different use cases for these tasks and the different pools used to complete each task.

Tools and methods for Machine Learning

In this session we will be reviewing a number of different ways to solve problems with machine learning to better understand what is required to be able to problem solve with Machine Learning. As part of this discussion we will focus on the data requirements for machine learning, to better understand the kinds of data needed for analysis.

We will also review several different tools for creating machine learning solutions in Azure, including Azure ML and Synapse.

No Machine Learning experience required

Implementing a Self-Service Power BI Solution

Providing an environment where business users are able to develop their own reports is a goal of many companies. Providing this environment in Power BI takes more than just pointing users to a data model, as the data environment needs to provide an environment where users can utilize design standards, understand which measures to use so that they will be able to do their own analysis. We will look at all of the steps needed to create detailed themes, templates, and supportive model designs. Learn everything you need to include to make non-Power BI experts look good and create meaningful analysis.

Diving into Dax: 7 things that will help you write better DAX and understand why

If you use Power BI you have written DAX. DAX can be complicated and understanding better how the DAX Engine works can help you write better performing DAX. There are a couple of third party tools which can help, so we will review Bravo, a relatively new free tool you may not know about and DAX Studio. We will use those tools when we review DAX Formatting with Bravo, context transition, different filtering methods and why you would use them, Calculated tables, USERELATIONSHIP() tricks, when to use VALUES, and validating performance with DaxStudio. After this session you will be able to incorporate some useful tools, write better performing DAX, measure the performance and better understand the DAX Engine in Power BI.

Power BI Data Governance

Developing a Power BI data governance environment combines technical elements with managerial skills. To effectively manage a Power BI Environment, you will need to develop and incorporate strategies for licensing, data security, source control, report distribution, consistent development standards, and model management. In this session we will review what technical elements you need to implement and the governance practices needed to ensure you have a robust scalable Power BI Environment.

Data Lake or Data Warehouse? Which one makes sense?

In this session we will explore data lakes and how you can use them as a data warehouse. We will also explore creating a traditional Data warehouse using Dedicated Pools in Azure Synapse and review which one makes sense given different environments.

Data storage and Usage in Microsoft Fabric

Microsoft Fabric has different storage and exploration for data which were not available in Synapse or Power BI. One Lake, Data Lakehouse, and SQL Endpoints are three different ways of organizing data that may provide a significant benefit to your environment. In this session we will explore these different storage options and the use cases for each. We will also review a new method for exploring data stored in One Lake, Data Wrangler. The demos will provide examples and since Microsoft Fabric is still in preview, you will be able to work through them later yourself.

Data Engineering in Microsoft Fabric

Microsoft Fabric includes a lot of different elements, including Data Engineering. Data Engineering includes notebooks, pipelines, lakehouses and data pipelines which you will have a better understanding of how they work together within fabric. In the demos for this session you will see what functionality these elements provide and how you can use them in your data solutions.

Introduction to Microsoft Fabric

Microsoft Fabric was introduced in May of 2023 and contains elements of Power BI, Synapse, and Machine Learning. In this session we will review the different components and focus on what elements can be used to architect a data solution, how fabric differs from other previous technologies and how it doesn't and why you would want to use it in your environment.