Large Language Models (LLMs) have gained significant prominence due to their diverse applications, ranging from conversational agents to code generation assistants. Given their increasing deployment in production environments, understanding and monitoring LLM behavior has become crucial for effective implementation and risk management.

Observability for LLMs goes beyond monitoring what prompts are sent to the model and what responses are received, it includes also monitoring the application making the call in a distributed system, and considering the wide range of options for using a Large Language Model from using cloud hosted versions to using local open models. Kubernetes is a common platform for deploying the apps and the LLMs

In this session we will explore how OpenTelemetry, the Open Source de facto tool for Logging, Monitoring and Tracing can be used on top of Kubernetes to keep an eye on applications and LLMs behavior. We will explore tracing calls, monitoring prompts and parameters and costs.