Running generative AI applications can be prohibitively expensive. This talk unravels the soaring costs of inference - providing real-time responses to user queries using large language models with billions or trillions of parameters.

We estimate the staggering costs to serve individual user queries by accessing massive models, and delve into the performance and cost challenges, from GPU hardware accelerators to latencies in running ChatGPT.

We explore the potential for improving resource efficiency and performance of running large-scale AI applications on Kubernetes and in cloud-native environments through GPU resource sharing, advanced scheduling, and dynamic batching.

We hope this talk will spur further discussion and collaboration within the CNCF community around taming the costs of deploying and scaling generative AI using cloud native technologies and best practices.