Generative AI is non-deterministic, and as an industry we are still learning how to use it. This makes observability especially important for AI applications. Beyond typical performance and usage telemetry, AI systems require capturing conversation data to ensure responsible behavior and meaningful results.
Conversation history, however, is verbose and often contains sensitive information, which introduces extra challenges and costs.

In this session, we’ll show you how to instrument your AI applications with OpenTelemetry following the Generative AI Semantic Conventions.

Through a live demo, we’ll demonstrate how to record conversation history alongside performance-related telemetry, and how to separate sensitive data for compliance and cost-saving purposes. We’ll also explore options for running evaluations and recording results based on telemetry data.