In TTChat, microservice observability faces two challenges: how to instrument microservices without code modification across diverse programming languages and frameworks, and how to connect logs, metrics and traces.

We will share our story of building an observability platform. We relied on client SDK initially, collecting 200TB data per day, and then transitioned to using eBPF for auto-instrumentation. We explored various approaches to connect the three pillars of observability. By leveraging the OpenTelemetry Collector and self-built exporters, we preserved meaningful observability data, particularly in connecting error traces and logs with metrics.

We will share those practical experiences and discuss why connecting eBPF data without context is difficult. We will not only showcase the key of different approaches but also delve into the challenges we encountered. Attendees can learn how to connect observability data and take away valuable lessons to avoid repeating our mistakes.