What is causing slow cold starts in Kubernetes-based LLM serving systems beyond model loading?
In Kubernetes-based LLM serving, we often blame cold starts entirely on model loading. In reality, a large part of the delay comes from GPU kernel compilation. Even with model weights cached, kernels are recompiled from scratch on every new pod, even on every pod restart
In this talk, we share a practical KServe-based solution that captures the compiled GPU kernels during the initial cold start, stores them as signed OCI artifacts, and caches them alongside the LLMInferenceService.
New pods can then load these pre-compiled kernels at runtime, skipping redundant compilation during scale-out. In our tests under burst traffic, this approach reduced vLLM pod readiness time by 30–70%, depending on model size and GPU type.
If you're an AI platform engineer who is running large language models on Kubernetes, you will learn how kernel-level caching can be integrated into your serving stack.