Short Description
Learn how to create ground truth test sets for Microsoft Fabric Data Agents using a practical framework from a Microsoft Fabric Community blog series. Discover how to measure accuracy, improve trust, and operationalize AI quality testing.

Session Abstract

Microsoft Fabric Data Agents can transform how users interact with data, but how do you know when an agent is providing accurate, trustworthy answers?

Based on a Microsoft Fabric Community blog series and real-world enterprise implementations, this session introduces a practical framework for creating and maintaining ground truth test sets for Fabric Data Agents. While many organizations focus on prompts and semantic models, far fewer establish a repeatable process for validating agent behavior and measuring answer quality over time.

We'll explore how to build business-driven test questions, define expected outcomes, evaluate response quality, and create repeatable validation processes that support production-ready AI solutions. Attendees will learn how ground truth testing can help identify regressions, improve stakeholder trust, and establish governance processes for enterprise AI workloads.

Using examples and lessons learned from the blog, we'll walk through a complete testing lifecycle, from identifying critical business questions through scoring, monitoring, and continuous improvement.

Whether you're experimenting with your first Data Agent or supporting enterprise-scale AI adoption, you'll leave with practical guidance, templates, and a proven methodology for measuring what matters most: the quality of the answers your users receive.

Learning Objectives

By the end of this session attendees will be able to:

Understand the purpose and value of ground truth testing
Create representative business-focused test sets
Define measurable success criteria for Data Agent responses
Identify accuracy, consistency, and completeness issues
Detect regressions when semantic models or agents change
Develop ongoing evaluation and governance processes
Improve stakeholder confidence in AI-powered analytics

Session Duration
- 45 minutes (35 minutes presentation, 10 minutes Q&A)

Session Format
- Educational presentation combining concepts, testing frameworks, implementation guidance, and light demo.

Technical Level
- Intermediate (Level 200)

Target Audience
- Architects, Fabric Administrators, BI Professionals, Analytics Engineers, Data Engineers, AI Practitioners, Governance Teams, and anyone responsible for deploying and supporting Fabric Data Agents.

Prerequisites
- Basic understanding of Microsoft Fabric, semantic models, or Data Agents is recommended but not required.

What Attendees Will Receive:
- Ground truth testing framework
- Test question design methodology
- Evaluation scoring concepts
- Governance recommendations
- Practical implementation guidance
- Real-world lessons learned from enterprise deployments

Conference Tracks
- Microsoft Fabric, Data & AI, Analytics, Governance, Business Intelligence, Architecture, Data Platform, and Enterprise AI.