Based on real production experience operating an observability-driven delivery platform on CNCF projects, this session shows how OpenTelemetry becomes a core primitive for progressive delivery, and how Argo Rollouts turns telemetry into safe, automated rollbacks.
Running dozens of Kubernetes clusters at 60M+ requests per minute, we hit gray failures: builds passed, pods stayed Running, liveness was green, yet real users hit errors. No health signal was scoped to the specific build.
We thread a single build_id from CI (GitHub Actions) into every OTel span, log, and metric. An Argo Rollouts AnalysisTemplate runs PromQL scoped to that build_id and aborts the canary automatically when error rate breaches the SLO.
Then we add a cognitive layer: an AI SRE agent reads the same build_id-scoped Loki and Prometheus signals via the Grafana MCP server, names the offending commit, and opens a PR a human merges. We share the trade-offs and lessons learned.