Session

Building a User-Facing Audit Log Archive with OpenTelemetry and DuckDB

Your OTEL traces already contain every business event your users care about: file uploads, payments, and document changes. This talk shows how to filter that data out of the infrastructure noise, store it in S3 or GCS as plain JSONL, and query it with DuckDB. Filtering down to audit events cuts storage by 92%; the DuckDB warm path is 2,200x faster than scanning raw JSONL. No new databases, no managed services, no per-query cost.

Tiered storage is the key architectural decision. Hot queries hit an in-memory cache; warm queries hit a denormalized DuckDB file (23ms); cold queries hit JSONL on object storage (single-digit seconds). Each tier serves a different query pattern without forcing everything through one system.

A single Protobuf schema ties the ingestion pipeline and search API together, so event classification stays consistent as the system evolves.


Audience: backend, platform, and DevOps engineers working with OpenTelemetry. Familiarity with OTEL concepts (spans, attributes) is helpful but not required.

Duration: 30–40 min. Talk includes code examples and benchmark data.

First delivery. No special requirements.

AJ Danelz

Golang enthusiasts | DevRel | Cloud Native | Streaming

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top