Session
When Your Multi Agent System Has to Speak Out Loud
WHAT
Imagine saying "find me a hotel in Rome for two, and something to do if it rains". From that one conversation you get a booking, a map and a day by day plan. We built that assistant.
A Copilot Studio orchestrator listens to the traveller and hands each request to the right specialist agent:
a hotel agent that books rooms in Dataverse through Power Automate
a local discovery agent that chains location and places lookups
a logistics agent for flights, weather and currency
a destination expert that runs in Azure AI Foundry with AI Search and Code Interpreter
We also built a second version: a speech to speech concierge on GPT Realtime over WebRTC that you can interrupt while it talks. This session shows both, running live.
WHY
Adding a microphone to a multi agent system changes more than the input box:
Nobody wants to hear six flights read out loud.
Every second of silence feels like something broke.
People interrupt.
The keys that call your agents must never reach the browser.
Most teams also get stuck on a question that sounds simple: should this agent live in Copilot Studio or in Foundry? Our answer was both, each doing what it does best, and we'll explain why.
HOW
We start with a live voice conversation and then show what's underneath it. You will see:
how the orchestrator picks an agent using nothing more than that agent's description
how a Foundry agent joins a Copilot Studio orchestrator as a connected agent
how agent events update a map and a quotation panel on screen, while a model shortens long answers into one spoken sentence
Then we switch to the realtime concierge and compare the two voice designs side by side: speed, interruptions, grounding and how easy each is to change. Everything we show is in public GitHub repositories.
WHO
Makers, developers and architects who build agents on the Microsoft stack. It suits people who want to add voice, and people deciding between low code and pro code for each agent. Knowing some Copilot Studio or Azure AI Foundry helps, but you don't need any voice experience.
KEY TAKEAWAYS
- A multi agent pattern where a low code orchestrator routes to specialists in Copilot Studio and Azure AI Foundry, and how to decide where each agent belongs.
- Why agent descriptions are the real routing logic, and how to write them so the orchestrator chains tools by itself.
- Two voice designs, a speech to text and text to speech pipeline versus realtime speech to speech, and when to choose each.
- Designing for the ear: speak the short answer and show the details on screen.
- How to keep every secret out of the browser with short lived tokens.
- The fallbacks that keep a live voice assistant working when one piece fails.
David Lorenzo
Microsoft MVP Business Apps | AI @ Automation Architect at Prodware
Adeje, Spain
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top