An AI assisstant that only answers questions is one thing. An agent that sends email, creates calender events and searches your private documents is another. When an agent can act, every design gap becomes a trust problem.
In this session I share lessons from building Aether, an open source agentic AI workspace. It answers from user's oen documents and acts in gmail, google calender and google drive by text and by voice also. Behind it sit 18 tools, an MCP client, a provider-agnostic LLM layer and a hybrid retrieval pipeline, deployed as three independent services.

This isn't a success story. It's what building it taught me about trust:

Enforce boundaries in the data layer, not the prompt. Documents are isolated per conversation in SQL. I'll also show a latent risk I found where the guard lived in Python instead.
Measure before you believe. On a labelled benchmark, hybrid search (vector + BM25 fusion) improved Hit@3 by 33%. The cross-encoder reranker everyone recommends made results worse. I published that result.
Silent failures are the dangerous ones. One bug made failed tool calls reach the model as successes. There was no crash and no log.
Actions need human gates. My agent could send email the moment the model decided to. Approval steps and audit logs belong in the design from day one.
Real-world access is regulated. Google's verification tiers for sensitive scopes show what production access to user data actually demands.
Avoid vendor lock-in. One interface across OpenAI, Anthropic, Gemini and Groq, with a mock provider so 159 tests run without API keys or data leaving the environment.

Attendees will leave with a practical checklist for building AI agents that organizations can trust with real data and real actions.