Jooho lee
Red Hat, Principal Software Engineer
Toronto, Canada
Actions
Jooho Lee is a Toronto-based technology professional with over a decade of experience at Red Hat, working across cloud-native platforms and AI. He began in middleware support and later moved into OpenShift consulting, helping customers design and operate Kubernetes platforms at scale. Today, he is on the OpenShift AI Model Serving team, integrating KServe and LLM-D into Open Data Hub to enable production-grade, Kubernetes-native deployment and operation of AI/ML models. He is also a KServe committer, contributing to the project’s model serving and LLM inference abstractions.
Badges
Area of Expertise
Topics
Reduce vLLM Pod Cold Starts with KServe Kernel Caching
What is causing slow cold starts in Kubernetes-based LLM serving systems beyond model loading?
In Kubernetes-based LLM serving, we often blame cold starts entirely on model loading. In reality, a large part of the delay comes from GPU kernel compilation. Even with model weights cached, kernels are recompiled from scratch on every new pod, even on every pod restart
In this talk, we share a practical KServe-based solution that captures the compiled GPU kernels during the initial cold start, stores them as signed OCI artifacts, and caches them alongside the LLMInferenceService.
New pods can then load these pre-compiled kernels at runtime, skipping redundant compilation during scale-out. In our tests under burst traffic, this approach reduced vLLM pod readiness time by 30–70%, depending on model size and GPU type.
If you're an AI platform engineer who is running large language models on Kubernetes, you will learn how kernel-level caching can be integrated into your serving stack.
Beyond Model Serving: Running Production LLMs on Kubernetes with KServe
Running an LLM on Kubernetes is easy. Running it in production is a different story.
Once you move beyond a demo, you face challenges that go far beyond serving a model: GPU scheduling, traffic spikes, streaming responses, autoscaling, model updates, and keeping latency low while efficiently using expensive GPU resources. Production LLM workloads require new approaches to scaling, routing, resource management, and platform integration.
In this session, KServe maintainers will share how the project has evolved beyond traditional model serving. We'll discuss how inference engines, AI gateways, autoscaling, distributed serving, and traffic management work together through Kubernetes-native APIs to simplify running LLM workloads at scale.
We'll share lessons learned from building and operating these capabilities, including practical deployment patterns and design decisions from real production environment
KServe 101: An Introduction to CNCF’s New Model Serving Project
KServe is a newly accepted CNCF project designed to simplify and standardize machine learning model serving on Kubernetes.
This lightning talk provides a concise introduction to KServe for those new to the project. I will cover what KServe is, what it provides, and its key building blocks—such as InferenceService, serving runtimes and llmisvc(llm-d) —giving attendees a clear mental model of how KServe works and where it fits in modern cloud-native ML platforms.
KServe and the Next Step for LLM Workloads: LLMInferenceService in Context
Traditional ML and LLM serving share the same needs on Kubernetes: consistent deployment APIs, reliable scaling, and an operational model teams can standardize and automate. This session gives an architecture-first overview of KServe, a Kubernetes-native model serving control plane for classic inference and LLM workloads (a CNCF incubating project since Nov 2025). We then explain why KServe introduced LLMInferenceService (LLM-D integration): to support LLM-focused serving patterns within the same KServe foundation, with clear responsibilities and a Kubernetes-native request flow. You’ll leave with a strong mental model of how KServe and LLMInferenceService fit together and how to approach adoption on real platforms. We close with a short demo installing KServe and deploying a minimal inference service (no benchmarking or tuning).
Project Lightning Talk + ContribFest + Maintainer Track: KubeCon + CloudNativeCon NA 2026 Sessionize Event Upcoming
KubeCon + CloudNativeCon North America 2026 Sessionize Event Upcoming
CNCF Toronto: 2026 Meetups User group Sessionize Event Upcoming
KCD Texas 2026 Sessionize Event
KCD Toronto 2026 Sessionize Event
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top