Deploying and managing large language models (LLMs) in production often presents significant challenges.By self-hosting LLMs on Kubernetes, organizations gain enhanced data privacy, flexibility in model training, and potential cost savings. This talk aims to enabling beginners by demystifying the process of self-hosting LLMs within the robust Kubernetes ecosystem.We will place a special emphasis on harnessing the capabilities of the Podman Desktop AI Lab extension to accelerate and simplify the development, deployment, and management of LLM workloads on Kubernetes.
Key topics will include:
- Strategically selecting and containerizing suitable open-source LLM models for optimal performance
- Crafting Kubernetes deployment manifests tailored for LLM workloads
- Provisioning and managing Kubernetes resources to meet the computational demands of LLMs
- Deep dive into leveraging the Podman Desktop AI Lab extension for streamlined LLM workflows on Kubernetes