Deploying an open source LLM on Kubernetes demands more than spinning up a single inference container. GPU scheduling, routing, and observability must work together. This lightning talk presents a condensed recording of a 15-minute deployment that turns a Kubernetes cluster with GPU nodes into a working inference platform: an open source model behind an OpenAI-compatible endpoint and a chatbot UI, all built from open source tools.

A declarative recipe composes GPU resource allocation, a Gateway API-based inference gateway, a scheduler, and Prometheus around an open source inference engine. AICR, an open source recipe tool, turns the composition into a Helm-based deployment bundle. It covers prerequisites, deployment order, and validation evidence mapped to Kubernetes AI Conformance requirements.

Attendees see how the recipe captures integration decisions, how the generated bundle makes them inspectable, and how to map deployment results to platform capabilities on their own cluster.