Cijo Thomas
Software Engineer - Observability at Apple
Seattle, Washington, United States
Actions
Cijo is a Software Engineer at Apple working on Observability. Previously, he worked on observability and telemetry solutions at Microsoft. He has been involved with OpenTelemetry since its inception and is a core maintainer of the OpenTelemetry .NET and Rust implementations. He is also an Approver for the OpenTelemetry Specification and OpenTelemetry Arrow projects.
Area of Expertise
Topics
The Invisible Tax: How Data Format Conversions Drive Up Telemetry Pipeline Costs
Telemetry signals traverse long pipelines before reaching observability backends. While enrichment, filtering, and redaction provide clear value, significant compute cost often comes from repeated conversion through different data formats.
Telemetry commonly flows through SDK formats, wire protocols, collector‑internal formats, and backend ingestion schemas. Each boundary introduces marshaling, unmarshalling and copying. These transformations add no new information, yet consume CPU and memory and scale linearly with volume—creating a hidden "transform tax" that compounds dramatically at terabyte scale.
This talk will share results from measuring instrumented OpenTelemetry SDK and Collector pipelines. We quantify compute spent on pure format conversion versus value‑generating processing and show how these costs grow with scale.
Attendees will learn about conversion costs and strategies to reduce waste: eliminating unnecessary translations, aligning pipeline representations, leveraging zero‑copy techniques, and minimizing transformation hops between pipeline stages. We also examine Apache Arrow‑based representations as one approach to reducing this overhead.
OpenTelemetry Metrics Just Got 25× Faster
OpenTelemetry's metrics performance has long been a sore point for high-throughput users. Recording a simple Counter with three attributes/labels cost around 50 nanoseconds per call - enough to show up in CPU profiles and rule OpenTelemetry out of the busiest code paths.
Pre-resolved metric handles are an old answer. Prometheus client libraries expose labelled-metric handles you can cache, and Windows Performance Counters have used the same pattern for decades. OpenTelemetry now offers it as a first-class API, called bound instruments, and on that same hot path, recording drops to under 2 nanoseconds - roughly 25× faster (Measured in OTel Rust Sdk)
This lightning talk shows where the speedup comes from, when to reach for it (high-frequency counters with a fixed, known attribute set), and - more importantly - when not to. Used the wrong way, the new fast path can be slower than the original.
Flexible by default, fast when you need it.
“Metrics That Lie”: Understanding OpenTelemetry’s Cardinality Capping and Its Implications
High-cardinality metrics just got safer with OpenTelemetry’s cardinality capping—SDKs won’t get OOM-killed anymore—but there’s a dangerous catch. When cardinality limits are hit, new attribute sets are folded into a special overflow attribute. The problem? Metric backends like Prometheus don’t treat this overflow attribute as special—it’s just another label—so queries can silently return incorrect results.
Imagine querying error counts for /endpoint1. Once the cap is hit, new errors for that endpoint get folded into overflow. A query like errors WHERE url="/endpoint1" might underreport errors—masking real issues.
This lightning talk explains how OTel’s cardinality capping works, how it can create hidden observability traps, how to detect when your queries may be incorrect, and strategies to avoid falling into this subtle but impactful pitfall.
The spec for cardinality capping is already stable, and support is rolling out across SDKs—now’s the time to understand its impact and prepare.
High-Volume Logging Without High Cost: Flight Recorder for OpenTelemetry Logs
Organizations face escalating costs from high-volume application logs—most never used. In response, teams aggressively filter logs, risking incident blind spots.
This talk introduces the Flight Recorder pattern for OTel Logs: instead of continuously exporting logs via OTLP, applications export them to ring buffers—ideally backed by OS-native mechanisms like Windows ETW or Linux Tracepoints, but also implementable in user space. Logs remain local, continuously overwritten until triggers—critical errors or operator requests—cause buffered logs to be exported via OTLP to OTel Collector, maintaining existing workflows.
Once triggered, the system also enters a temporary streaming mode, exporting all new logs continuously for a brief period. This approach eliminates unnecessary logging costs while ensuring logs are readily available when it matters. We’ll present a design and working prototype that shows how this model delivers substantial cost savings without sacrificing observability.
Beyond OTLP: Unlocking the potential of OS-native tracing
OTLP has become the de-facto standard for exporting telemetry in OpenTelemetry, but its reliance on networking (TCP) and batching introduces challenges-CPU contention, memory overhead, potential data loss during crashes, and complex retry and back-pressure handling.
This session explores a powerful alternative: leveraging OS-native tracing mechanisms—ETW (Event Tracing for Windows) and Linux user_events—to export telemetry synchronously via kernel-backed buffers. These mechanisms eliminate batching, network stack overhead, and retry logic, offering exceptional performance and reliability. Moreover, they are naturally suited for dynamic, on-demand telemetry, allowing telemetry to be enabled or disabled without restarts or redeployments—ideal for large-scale systems needing minimal overhead when telemetry is off.
Attendees will learn how ETW and user_events work, how they integrate with OpenTelemetry (without needing re-instrumentation), and how event listeners (e.g., the OTel Collector) can forward data via OTLP—preserving existing pipelines while gaining advanced observability features like capturing call stacks or triggering actions (e.g., process dumps) on specific events.
Behind the Code: Design Choices in OpenTelemetry .NET Metrics API and SDK
In the field of observability, metrics play a key role in monitoring system behavior and identifying potential issues. A reliable, high-performing, and always-running Metrics SDK is essential for monitoring the health of software systems.
This session will explore the OpenTelemetry .NET Metrics SDK, highlighting its design principles aimed at performance, predictable and bounded memory usage, and memory efficiency. We will delve into the engineering challenges faced by the team and the various design trade-offs considered. We'll briefly cover performance testing techniques.
A notable feature of the SDK is its ability to record measurements in as low as 10 nanoseconds, and up to just a few hundred nanoseconds, all with zero heap allocation. Moreover, the SDK includes memory limit safeguards to ensure bounded memory usage, regardless of the input, thereby ensuring that the telemetry system itself does not become an attack vector for security threats.
Join us for an overview of how OpenTelemetry .NET Metrics strives to meet its performance and efficiency objectives. We'll conclude the session with actionable insights for end-users to effectively use OpenTelemetry .NET Metrics!
CNCF-Hosted Co-located Events North America 2026 Sessionize Event Upcoming
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top