If your multi-agent pipeline forces users to watch a loading spinner while it thinks, you have already lost them. But if your agent hallucinates a high-pressure gas line on a city street, you have a much bigger problem.

This session strips away the "Hello, World" chatbot boilerplate to walk through the architecture of The Invisible City, a real-time, see-through AR visualization of underground municipal infrastructure. We will dissect how to pair Gemini’s multimodal vision capabilities with a Next.js and FastAPI stack to build event-driven agents that actually interact with the physical world.

More importantly, we will cover where AI fails. We will look at how to design Human-on-the-Loop (HOTL) UX, stream incremental progress using Server-Sent Events (SSE) so users aren't left in the dark, and implement the deterministic fallbacks required when a generative model decides to render a giraffe instead of an APWA-coded utility line.

Key Takeaways:

Streaming Multi-Agent UX: How to wire Server-Sent Events (SSE) into asynchronous FastAPI streams so users see incremental pipeline progress.

Multimodal Spatial Alignment: Translating Gemini segmentation masks and bounding boxes into APWA-compliant transparent overlays.

Calibrated Trust & Fallbacks: Designing deterministic guardrails and HOTL interfaces for spatial data where hallucinations carry real-world physical risks.

related:
https://medium.com/google-developer-experts/google-glass-projected-data-gemini-generates-reality-a769ccf288eb

As a GDE for Cloud AI/ML, Udacity instructor, and AI Director, my goal is to push developers past brittle, prompt-based demos into governed, multi-agent systems that survive contact with reality. This is a code-first, full-stack engineering story tailored for builders who care about architectural tradeoffs, performant UI, and responsible AI deployment. I will include a live-running demo of the browser UI rendering the pipeline.