Anyone who can open a support ticket can now talk to something that reads all of production. I built it: an autonomous AI agent living inside the prod cluster, woken by a ticket webhook, investigating 20 data sources (OpenSearch, RDS, BigQuery, logs, Kubernetes) and replying on the ticket. Always read-only.

I tried every open-source investigation agent I could find; none were safe to give production access. So I built my own - and the hard part was never the investigating, it was the defense.

You'll walk away from the session with: the read-only-by-construction pattern for giving an agent production access (no write tool to call); the guardrail stack - locked-down tool surface, network and RBAC limits, prompt-injection defenses - that treats every ticket as untrusted input; how to stream an LLM's own traces into your APM (and the one NetworkPolicy that silently broke it); and an honest look at where it saved on-call - and where it answered with full confidence and was wrong.