In the rapidly evolving generative AI landscape, KServe has emerged as a pivotal platform for deploying and managing LLMs at scale. KServe simplifies deploying ML models on Kubernetes, but there’s so much more to the story than predictor pods and YAML files. With its newly expanded capabilities, KServe is ready to host the next generation of AI workloads, including LLMs and other generative AI applications.
As both maintainers of KServe and daily practitioners running it in Bloomberg’s clusters, we bring firsthand insights into how users utilize KServe to deploy advanced LLM features in production across hybrid environments. This session will delve into KServe's latest features tailored for generative AI. We will offer insights into its enhanced serving runtimes, scalability improvements, and integration strategies. Attendees will gain practical knowledge about deploying and scaling generative models using KServe, informed by real-world experiences and the lessons we’ve learned.