Build a Lakehouse in an Hour with CF.Cumulus

In today’s data-driven world, fast and efficient data platform delivery is crucial for staying ahead of the competition. Join us for an exclusive session hosted by Cloud Formations, where we will demonstrate how our CF.Cumulus product can help you build a metadata-driven Lakehouse using Microsoft cloud native technologies. With CF.Cumulus, leverage Azure Data Factory, Azure Databricks, Azure Synapse Analytics, or Microsoft Fabric to streamline and optimize your data infrastructure.

Discover how Cloud Formations can help you simplify and overcome common obstacles such as fragmented data ingestion, change data capture, and orchestration scalability using our proven best practices. Learn how to leverage automation, open-standards, and seamless cloud integration to accelerate time-to-insight with minimal technical debt. This session is perfect for techies and data leaders seeking to streamline their cloud data platform delivery while maintaining cost control and operational resilience. In summary, unlock the potential to build a Lakehouse in a day with Cloud Formations’ CF.Cumulus product accelerator.

How Can Microsoft Fabric Have An Impact On Your Business

All this talk about Data-Ware-Lake-Delta-Beach-House-Lakes (or some combination of that) and Data, Yarn, Fabric integration, everything has got a bit… Meshy! Yes, my friends. The beat of the technology drum is certainly relentless. And with no limits cloud scale and huge innovations from the biggest brains. Two years, it seems, has become the benchmark for tools to live and die by. Reach three years and you almost have a mature product. That said, Microsoft Fabric, the latest offering from global software giant is no exception. But what does this mean for the real world. For the data analysts, engineers and scientists that need to continue answering everyday problems to inform business decisions. In this session we will firmly ignore the hype and focus on the reality. With the pragmatic view of an experienced architect. The problem of gaining insights from our data hasn’t changed. So, what does this mean if implemented using Microsoft Fabric. What, why and how is the tooling going to change our daily deliverables in the short term, medium and long term. Join me for these answers and more as we explore the impact of Microsoft Fabric-Server, erm, Power. Resource. Thing!

Building Near Real-time Data Solutions in Microsoft Azure & Fabric

The velocity of data is getting faster across many industries, fuelled by the business demand to gain insights and value from sources in near real-time. This necessity is then allowing decision makers to pivot and ultimately stay ahead of the competition. Furthermore, the growth of the internet of things and ‘smart’ devices now means the volume of that high velocity data has exploded. Meeting this demand requires new concepts and new designs for data/solution architects, with high throughput ingestion endpoints and query stream tools that can perform aggregations ‘on the fly’.

In this session, we will address the above head on. Discussing and designing architectures that can scale and burst for high throughput events. Querying using both SQL and KQL to blend stream and batch data feeds for downstream reporting.

As a platform, in Azure we’ll explore Event Hub and Stream Analytics to ingest and handle that initial data stream. Before applying the same patterns to other resources in Microsoft Fabric with and Event Handler and Real-time Dashboards through the Event House. Understanding the patterns to apply as an architect vs the tooling available for delivery.

Administering Microsoft Fabric: Capacities, Workspaces, and Domains

In this session, we will explore the administration of Microsoft Fabric, with a focus on the organisation and management of data storage/compute through the configuration of capacities, domains, data products, environments and workspaces. We will discuss the application of data mesh and data fabric concepts in the context of Microsoft Fabric capabilities, including organising data products for effective delivery to business users. Also, considering how Microsoft Fabric compares to the practices and technical standards we’ve applied in Microsoft Azure over the years, making that shift from PaaS to SaaS, in theory and maybe in practical.

Additionally, we will explore how to use domains and separate workspaces to serve reports to business users, providing them with the information they need to make informed decisions. Aligning industry governance standards to Microsoft Fabric features and access controls. Join us to learn how to effectively administer Microsoft Fabric beyond the simple Workspaces inherited from Power BI

Fast-Track Your Lakehouse Build with a Metadata Framework

In today’s data-driven world, fast and efficient data platform delivery is crucial for staying ahead of the competition. Join me for a dynamic session that demonstrates how to build a metadata-driven Lakehouse with Microsoft cloud native technologies. Using your preferred compute and storage resources, Azure Data Factory, Azure Databricks, Azure Synapse Analytics or Microsoft Fabric.

Discover how to simplify and overcome common obstacles such as fragmented data ingestion, change data capture, and orchestration scalability using proven best practices. Learn how to leverage automation, open-standards, and seamless cloud integration to accelerate time-to-insight with minimal technical debt. This session is perfect for techies and data leaders alike, seeking to streamline their cloud data platform delivery while maintaining cost control and operational resilience. In summary, unlock the potential to build a Lakehouse in a day by using an open-source metadata driven product accelerator.

In today’s data-driven world, fast and efficient data platform delivery is crucial for staying ahead of the competition. Join me for a dynamic session that demonstrates how to build a metadata-driven Lakehouse with Microsoft cloud native technologies. Using your preferred compute and storage resources, Azure Data Factory, Azure Databricks, Azure Synapse Analytics or Microsoft Fabric.

Harnessing the Power of Apache Spark & Delta Lake in the Microsoft Data Ecosystem

Apache Spark is a powerful distributed compute engine that has become an industry leading solution for data processing. In this session we’ll firstly introduce Apache Spark, exploring its core concepts and capabilities. Then we will discuss how Apache Spark can be implemented using various Microsoft products, including Azure Databricks, Azure Data Factory, Azure Synapse Analytics, and Microsoft Fabric, to build robust, scalable data processes to perform advanced analytics, and drive business insights.

We’ll then explore combining Apache Spark with the open standard Delta Lake to offer a comprehensive solution that addresses both compute and storage aspects, allowing us to create a complete cloud-native data platform. Delta Lake enhances Spark's capabilities by providing ACID transactions, scalable metadata handling for both streaming and batch workloads. This integration ensures data reliability and consistency while enabling efficient large-scale data operations. Together, Apache Spark and Delta Lake facilitate the construction of resilient data pipelines, allowing businesses to leverage their data assets fully and achieve seamless data integration, transformation, and analytics within the Microsoft data ecosystem.

An Introduction to SQL – ANSI, Transactional and Spark Flavoured

This session will provide a foundational introduction to SQL, tailored specifically for new data engineers working within the Microsoft ecosystem of data platform tools. We will begin with the very basics of the ANSI standard, ensuring that participants gain a solid understanding of SQL as a language. We will explore the essential concepts of SQL, including data retrieval, modification, and manipulation, all while becoming familiar with the standard syntax and commands that form the basis of SQL across various platforms.

As we progress, we will delve into the specifics of T-SQL and Spark SQL, exploring how these dialects have evolved within the Microsoft SQL family of products and Apache Spark environments. Attendees will gain insights into the unique features and capabilities of each variant, learning how to effectively leverage T-SQL for powerful data management and analysis within SQL Server, as well as how to harness the scalability and performance of Spark SQL for big data processing in Spark implementations. By the end of this session, participants will have the skills and knowledge to confidently structure and query data with the language.

Building a Microsoft Cloud Analytics Platform End-to-End

In today's rapidly evolving data landscape, staying ahead of the curve is essential for data professionals. This talk will explore the latest advancements in building an end-to-end Microsoft focused cloud analytics platform, leveraging all the technologies and industry best practices. Delving into the patterns and capabilities of both Microsoft Fabric and Microsoft Azure.

No longer can we rely on products matured over a decade to deliver all our solution requirements. Today, data platform architectures designed with best intentions and known design patterns can go out of date within months. That said, is there now a set of core components we can utilise in the Microsoft cloud to ingest, curation and deliver insights from our data? When does ETL become ELT? When is IaaS better than PaaS, and what SaaS products can help? Do we need to consider scaling up or scaling out? And should we start making cost the primary factor for choosing certain technologies?

In this session we'll explore the answers to all these questions and more from an architect’s viewpoint. Based on real world experience let’s think about just how far the breadth of our knowledge now needs to reach when starting from nothing and building a complete Microsoft cloud analytics platform.

Data Integration Pipelines – The Fundamentals

Join us for this fundamentals session where we will cover the essentials of data platform orchestration using Microsoft's suite of tools, including Azure Data Factory, Synapse Analytics and Microsoft Fabric. We will learn to construct control flow and data flow components, developing end-to-end processing pipelines. Starting with the basics, we will explore how these cloud-native resources have evolved and build pipelines that ingest data from various sources, transform this data, and make it available to consumers.

Throughout the session, we will delve into the integration pipeline tools within a highly scalable cloud-native architecture, addressing key aspects such as triggering, monitoring, dynamic pipeline content, and CI/CD practices. This session will provide a clear understanding of data ingestion, integration, and orchestration, equipping attendees with the knowledge to implement these practices in their roles as data professionals. By the end of the session, participants will have gained the foundational skills required to apply their new expertise in Azure Data Factory and Microsoft Fabric to real-world scenarios.

An Introduction to Delta Lake and The Lakehouse

Once upon a time, there was a data warehouse, and it lived happily as a set of tables within our relational database management system (RDMS) called Microsoft SQL Server. The data warehouse had three children known as extract, transform, and load. One day a blue/azure coloured cloud appeared overhead, and it started to rain. The data warehouse got wet and was never the same again! Or was it? Spoiler alert, the data warehouse is the same, still happy, and well, it just evolved and moved from its RDMS home to a new home in the cloud. The end!

In this session, we'll look at the evolution of the data warehouse and understand how we can now deliver the same data engineering concepts for our solutions on the Microsoft cloud platform using the open-source Delta.io standard, in Azure and Fabric. We'll introduce the standard (originally developed by Databricks) and then explore the implications it has for our next-generation cloud data warehouse.

The original data warehouse set of tables remain, but now they are delivered using the cloud-native Delta Lake technology with distributed storage/compute as standard. Delta.io gives us those much-needed ACID properties over our data lakes meaning our data warehouse understanding can move to the cloud and is made easier within Azure. The data warehouse just grew up and became a Delta Lake-House.

An Evolution of Cloud Data Architectures - Lambda, Kappa, Delta, Mesh & Fabric

How has advancements in highly scalable cloud native technology influenced the design principles we apply when building data platform solutions? Are we designing for just speed and batch layers or do we what more from our platforms, and who says these patterns must be delivered exclusively?

Let’s disrupt the theory and consider the practical application of all things Microsoft now has to offer, where concepts, patterns, and best practice meet/clash with technology. Can we now utilise cloud technology to build architectures that cater for lambda, kappa, and mesh concepts in a complete stack of services? And should we be considering a solution that offers all these principals in a nirvana of data insight high scalable, decoupled perfection? Lastly, how does Data Fabric as a concept fit with Microsoft Fabric as a product and should we decentralise everything as suggested by the data mesh!?

In this session we’ll explore the answer to all these questions and more in a thought provoking, argument generating look at the challenges every data platform engineers/architects face.