Supercharging Python in Microsoft Fabric: From Notebooks to Production-Grade Data Solutions

Python notebooks in Microsoft Fabric aren't just for data exploration - they're a gateway to building enterprise-grade data solutions. This session shows you how to elevate your Python development in Fabric from experimental notebooks to production-ready, maintainable code assets.

The session will provide practical demonstrations and real-world examples of:

- Best practices for structuring Python code in Fabric for reusability and maintainability
- How to implement version control and CI/CD pipelines for your Fabric Python assets
- Techniques for adding monitoring, logging, and observability to your Python code
- Methods for ensuring code quality through testing and documentation
- Tools and approaches that maximize developer productivity in the Fabric environment

Whether you're a Python developer new to Fabric or an experienced data engineer looking to optimize your Python workflows, you'll gain practical insights for building more robust, maintainable, and scalable solutions.

Practical application of Database Templates in Azure Synapse Analytics

Database Templates are a relatively new feature added to Azure Synapse Analytics.

They are provide a layer of abstraction over your Azure Data Lake Storage giving you an opportunity to build a semantic model.

They promise many benefits such as schema enforcement, discoverability and Power BI integration.

Barry will demonstrate these benefits through an end to end worked example. He will achieve this by using Database Templates to specify a semantic model, leveraging other tools in Azure Synapse Analytics to populate it, before consuming the Database Template in Power BI.

Power BI by Design: Creating Reports Users Can't Live Without

Drawing from real-world experience implementing Power BI across numerous organisations and domains, this session reveals how to create Power BI reports that seamlessly blend technical sophistication with exceptional user experience to create reports that users can't live without.

The session covers:

- Practical methods for user engagement throughout the design and development phases
- Strategies for rapid prototyping and iterative improvement
- Building semantic models that reflect real business language and processes
- Techniques for incorporating accessibility and branded design principles
- Approaches for empowering users to own and evolve their reports
- Future proofing your semantic layer by preparing it for generative AI integration

Discover how empathy-driven design principles can transform technically sound reports into indispensable business tools that users genuinely love to use.

Whether you're a BI developer, data analyst, or solution architect, you'll leave with actionable strategies for creating Power BI solutions that achieve both technical excellence and widespread user adoption.

Beyond the Dashboard Graveyard: Building Data Products That Actually Deliver Value

In many organisations, data teams are caught in a frustrating cycle—building reports, dashboards and analytics that consume significant resources but fail to deliver their promised value. Industry research suggests that up to 85% of data projects fail to meet their objectives, representing millions in wasted investment.

The symptoms are everywhere: dashboards nobody looks at, models that never reach production, and analytics that don't drive decisions. The gap between technical implementation and actual business impact isn't just disappointing—it's existential for data teams facing budget scrutiny and diminishing stakeholder trust.

In this practical session, you'll learn how leading organisations are applying product thinking to data initiatives, dramatically improving adoption rates and business impact. Drawing on 30 years of experience helping organisations transform their approach to data products, Barry will provide a proven framework that has enabled 10X returns on data investments.

Whether you're a data leader struggling with ROI, an analyst whose work isn't getting traction, or a business stakeholder trying to get more value from data investments, you'll leave with practical tools to immediately improve how you conceive, design, and deliver data products that drive real business outcomes.

Turbo charge your Data Engineering and AI workflows with Microsoft Fabric

Join Barry for an insightful demonstration of a Data Mesh-inspired architecture leveraging a comprehensive suite of features in Microsoft Fabric. This session showcases a complete, end-to-end data engineering and data science experience in Fabric using the iconic Titanic dataset to build a machine learning model predicting passenger survival.

Starting with data ingestion and exploratory data analysis, Barry will guide you through the essential stages of data cleansing, wrangling, and semantic model creation. The session will culminate in training, evaluating, and deploying a robust ML model, illustrating the practical application of key DataOps principles throughout the lifecycle.

Discover how Microsoft Fabric empowers you to create impactful data products, enabling incremental development of your data capabilities in a "microservices for data" architecture. Don't miss this opportunity to learn how to harness the power of Microsoft Fabric to build scalable, efficient, and innovative data solutions.

Key Fabric features in demo: task flows, lakehouse, notebooks (key packages: MLflow, Sempy "semantic link", AzureLogHandler, Great Expectations), pipelines (includes new Teams activity), semantic model, report, experiment, model, environment, source control.

Updated / improved version of demo and talk given at SQLbits 2024, overall score 8.38 out of 9. Comments: "Brilliant session, really enjoyed it and learnt a lot. Liked business context at the start.", "Very nice demo!", "Brilliant demos and easy to follow.", "A really informative session about Fabric", "A good structure of presentation."

Should you be worried about your Software Supply Chain?

Your software supply chain should already be on your risk register. This session will provide a holistic view of software supply chain risk, using real life examples to bring the threats to life.

Having established the landscape, endjin will then demonstrate how they are adapting existing DevOps processes to produce Software Bill of Materials (SBOMs) in order to protect their portfolio of 50+ open source projects from this emerging threat.

In doing so, they will highlight the standards (OpenChain), frameworks (SLSA) and tools that can be used to fast track creation of a foundation for license compliance, monitoring of supply chain vulnerabilities, and establishing security procedures as required by Article 32 of GDPR.

Modern DataOps In The Azure Cloud

Modern data platforms deserve the love and attention that web applications get when it comes to modern DevOps approaches. Data platforms often lack the same rigor and formalization of DevOps processes, such as little to non-existent test coverage, inconsistent and sporadic development and collaboration techniques, and a less efficient or formalized feedback loop when addressing end-user issues or ensuring quality throughout all areas of the solution. With the transition to the cloud, new practices are necessarily having to be adopted to keep delivering high-quality solutions.

Microsoft Fabric and Data Mesh - a perfect fit?

Microsoft Fabric has been influenced by Data Mesh which is Zhamak Dehghani's thought leadership about how to "deliver data-driven value at scale". In this presentation, Barry will provide an overview of Data Mesh, how Microsoft Fabric measures up to Data Mesh principles and provide recommendations about how Fabric can be used to deliver a Data Mesh inspired data vision for data and analytics in your organisation.

DataOps: How to Deliver Data Faster and Better with Microsoft Cloud

DataOps is "DevOps for data". It helps data teams improve the quality, speed, and security of data delivery, using cloud-based tools and practices. DataOps is essential for real-world data solutions in production. In this session, you will learn how to use DataOps to build and manage a modern data platform in the Microsoft Cloud, with technologies like Azure Synapse Analytics and Microsoft Fabric. You will also learn how to use various services and technologies to enhance your data platform, covering topics such as:

- Build & release management: automate and streamline your data pipelines and deployments.
- Versioning: track and manage changes to your data and code.
- Validation: ensure the accuracy and consistency of your data.
- Testing: verify the functionality and performance of your data applications.
- Monitoring, observability and alerting: measure, troubleshoot and escalate your data operations and outcomes.
- Developer experience: enable collaboration and productivity among your data team members.
- Infrastructure deployment: provision and configure your data resources and environments.
- Security and governance: protect and control your data assets and access.

Join us to discover the tools, techniques, and processes we use to provide secure, reliable, cost-effective data solutions in the Cloud.

How to create a high performance data team: lessons learned from the field

There are many factors that influence the ability for data and analytics teams to achieve their full potential. With most organisations seeking to become data driven, how can they build a data and analytics team that is able to meet increasing demands in an agile, cost effective and sustainable way?

In this session, Barry will explore the factors that lead to success (and the pitfalls to avoid). He will use real world examples to bring it to life based on his experience delivering data intensive, high business impact projects over the last 25 years.

Fake it 'til you make it: generating production quality test data at scale

Many organisations provide digital products or services that need to handle personally identifiable information.

The challenge is providing product and engineering teams with a sufficient volume of realistic looking synthetic data to enable them to design, develop and test their solutions.

Barry will present open source tools and open data sources that can be used to tackle this challenge, and then demo this in action to generate thousands of synthetic customers.

He will describe how this approach can be used to build better products, to test products using production quality data at production scale, and embed data quality and best practice information security practices in your engineering processes.

DataOps on Microsoft Fabric and Azure

DataOps is "DevOps for data". It helps data teams improve the quality, speed, and security of data delivery, using cloud-based tools and practices. DataOps is essential for real-world data solutions in production. In this session, you will gain an overview of how to use DataOps to architect, build and manage a modern cloud native data platform, using Microsoft Fabric and Azure.

Practical demos will be provided in the following areas:
- Architecture: applying principles such as "data as a product" to shape the solution.
- Validation: ensure the accuracy and consistency of your data.
- Testing: verify the functionality and performance of your data applications.
- Monitoring, observability and alerting: measure, troubleshoot and escalate your data operations and outcomes.
- Infrastructure deployment: provision and configure your data resources and environments.

Use this session as a springboard to discover the tools, techniques, and processes we use to provide secure, reliable, cost-effective data solutions in the Cloud.

How to deliver value from data in a rapid and safe manner using Microsoft Fabric and Azure. We will provide an overview of the principles backed up with practical examples.

Microsoft Fabric and Data Mesh: Revolutionizing Data and Analytics

Discover how Microsoft Fabric, influenced by the pioneering Data Mesh framework, can transform your data and analytics strategy.

In this insightful session, Barry will start with an introduction to Data Mesh, explaining Zhamak Dehghani's vision for delivering data-driven value at scale. He will then explore the features and capabilities of Microsoft Fabric that align with Data Mesh principles, showcasing how to create a scalable, data-driven ecosystem.

Throughout the talk, Barry will provide practical examples of how you can use Fabric to create reusable "data products" and promote democratisation of data and analytics by enabling "domain ownership" across your organization. He will also demo DataOps capabilities in Fabric that enable data and analytics capabilities to be scaled up safely by applying "federated computational governance".

Join this session to see how Microsoft Fabric can enable you to re-boot your data strategy to build innovative data solutions, promote collaboration, and scale up your data and analytics infrastructure efficiently and safely. Don’t miss this opportunity to learn how to harness the power of Microsoft Fabric to deliver impactful data-driven value.

This is an extended version of 20 minute talk I provided at SQLbits 2024, This was very well attended, achieving an overall score of 8.29 out of 9. Feedback "Really well delivered session, good pace, easy to follow, very informative, great content covered". By extending to 60 minutes, it will allow me to address the only suggestion for improvement in feedback: "Make it a longer session with examples".

I also presented on this topic to a packed room at the "London Fabric User Group" in February 2024.

I've produced a series of blogs around this topic in a series of blogs: for example: https://endjin.com/blog/2023/08/how-does-microsoft-fabric-measure-up-to-data-mesh. This has been mentioned / discussed in 2 episodes of the Azure Data Community Round Table "Tales From The Field" webcast: https://www.youtube.com/@Tales-from-the-Field.

Spark Unplugged: How In-Process Analytics Is Making Distributed Computing An Expensive Investment

The data landscape is experiencing a fundamental shift. For years, we've been told that serious data work demands distributed computing, cloud infrastructure, and complex pipelines. But what if the most powerful analytics engine is already sitting on your desk?

In this provocative, insight-packed session, we'll challenge the conventional wisdom around big data processing by exploring the emerging "data singularity" - the point where single-node computing power is outpacing the growth of most analytical datasets. We'll demonstrate how tools like DuckDB and Polars are revolutionizing analytics by bringing analytical capabilities directly into your application process, eliminating overhead, delivering mind-blowing performance and turbo charging your "inner development loop".

You'll learn how these in-process engines can process millions of rows on your laptop, often outperforming distributed systems like Spark while dramatically reducing complexity, cost, and carbon footprint. We'll share practical code examples showing how to implement these tools in your workflows, with special focus on integrating into your Databricks or Microsoft Fabric environment.

This session is perfect for:

* Data engineers tired of the overhead incurred working with distributed systems
* Data scientists seeking faster iteration cycles
* Data leaders looking to minimise total cost of ownership and accelerate time to value
* Anyone interested in the future direction of data processing

Walk away with a completely fresh perspective on data architecture, practical techniques to implement tomorrow, and perhaps a nagging question: Have we been overengineering our data solutions all along?

Based on a series of blogs: https://endjin.com/blog/2025/04/duckdb-rise-of-in-process-analytics-understanding-data-singularity

No-Compromise Data Apps: Why Streamlit is the Missing Piece in Your Analytics Stack

Ever found yourself wrestling with Power BI limitations or implementing complex workarounds for what should be simple user requests? Enter Streamlit – the Python package that's transforming how we deliver value from data.

In this session, you'll discover how Streamlit enables you to build beautiful, interactive data applications in minutes, not months – all without front-end development expertise. We'll showcase real-world applications we've built for clients that let users manipulate model parameters and instantly visualize impacts, delivering a level of engagement traditional BI tools simply can't match.

By the end of this talk, you'll have everything you need to create your first Streamlit app, along with practical guidance for taking your prototypes from desktop to cloud deployment. See firsthand how this transformative technology is helping organizations move users from passive dashboard consumers to active participants in the data experience.
Join us to unlock a whole new dimension of data value that puts the power of interactive modeling directly in your users' hands.