Session
Rethinking Test Automation: Ship Your Copilot Studio Agents with Confidence
Your agent worked in the test pane yesterday. Today someone changed a prompt, added a knowledge source or connected a new tool. Does it still work? With classic test automation you'd compare the output against an expected result. With agents that doesn't work. The same question gets a different answer each time, and both answers can be correct.
Most teams fall back on manual testing: a few people chat with the agent before every release and hope for the best. That approach is slow, can't be repeated reliably and doesn't scale once you have more than a handful of agents.
In this session I'll show a practical way to test Copilot Studio agents that fits how agents actually behave. We'll build test sets from scratch, from knowledge sources and from real conversations, including multi-turn scenarios. Then we'll pick the right way to grade them, from exact and keyword match through meaning comparison and tool use checks to custom graders with your own rubric. Finally, we'll put evaluation into your ALM process so a pipeline runs the tests and stops a release when quality drops. I'll also cover where the Copilot Agent Kit adds value, and the current limits and costs you should plan for.
Expect mostly live demos and lessons from real projects.
Key takeaways:
* How to define "good enough" for an agent that never answers the same way twice
* How to build test sets that reflect real use, not just the happy path
* Which test method fits which kind of check, and when you need a custom grader
* A reusable pipeline pattern that gates deployments on evaluation results
* When the built-in agent evaluation is enough and when to add the Copilot Agent Kit
* The current limits and credit costs, so your test strategy holds up in production
Holger Imbery
Microsoft MVP Business Applications - Copilot Studio & MCT & PowerPlatform | Azure | Dynamics - principal architect @NTTData Inc.
Berlin, Germany
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top