You've likely heard a lot about AI by now. I was in the same boat, believing my encounters with AI would be confined to ChatGPT and GitHub Copilot. However, life had different plans for me. Working at a company that provides tools for managing cloud infrastructure, I was faced with a challenge from our customers. They wished to run Open-Source Large Language Models (LLMs) in a serverless environment, balancing cost-effectiveness with performance.

In this talk, I aim to share the insights and learnings from this journey. I will outline three different strategies we proposed to our clients, discussing the advantages and drawbacks of each, alongside the ultimate solution we adopted. Additionally, I'll talk about the challenges we faced and the strategies we employed to solve them from, all from my perspective as someone new to AI.