Modern telemetry systems often struggle not because of missing features, but because of hidden constraints in how data is buffered, scheduled, and moved through the system. This session explores the practical realities of building a telemetry data plane that must operate under extreme throughput, tight latency budgets, and strict resource limits.

Using real-world experience from developing a high-performance open source telemetry agent, we’ll examine how design tradeoffs around buffering, concurrency, and I/O shape system behavior at scale. Topics include user-space serialization strategies, adaptive buffering models, memory-mapped persistence, and multithreaded I/O coordination, along with how these choices interact with core Linux primitives such as epoll, asynchronous I/O, and zero-copy techniques.

Rather than focusing on APIs or products, this talk dives into the mechanics and constraints that determine whether a telemetry system remains predictable under load. The discussion is grounded in production lessons learned from operating at billions of events per minute and highlights patterns that apply broadly to collectors, agents, and streaming systems.