Observability is your safety net - but at scale, it often fails first. As our Kubernetes platform grew, rising traffic produced more telemetry, overwhelming our stack and leaving us blind during incidents. High-cardinality metrics exhausted memory, and ingestion brownouts became recurring nightmares. We didn't have too much data; we had too little control.
In this talk, we dissect the concrete failure modes and show how we rebuilt observability by treating telemetry pipelines as first-class distributed systems with OpenTelemetry, Prometheus, Loki, and Tempo.

We will walk through production fixes:

- Active Traffic Shaping: OTel Collectors for batching and tail sampling.
- Defusing Cardinality Bombs: Prometheus recording rules to stabilise memory.
- Back pressure & Limits: Surviving 10x traffic spikes.

This isn't a tool comparison. It's a blueprint for building observability you can reason about under stress, so your monitoring doesn't become the “next outage” you're explaining.