Event-Driven ML: Scaling Inference Workloads on Kubernetes with KEDA

Machine learning workloads break the assumptions that Kubernetes' default autoscaling was built for. Standard HPA scales on CPU and memory, but ML inference is GPU-bound, often bursty, and frequently queue-driven, meaning your pods can look "healthy" on CPU while requests pile up and latency climbs.

In this talk, we'll explore how KEDA (Kubernetes Event-Driven Autoscaling) solves this gap by letting you scale on signals that actually matter for ML — queue depth, GPU utilization, inference latency, and custom Prometheus metrics — while also enabling scale-to-zero for expensive GPU resources.

We'll walk through:
1. Why traditional HPA falls short for ML inference and batch workloads
2. KEDA fundamentals: ScaledObjects vs ScaledJobs, and when to use each
3. A practical example: scaling an inference server (e.g. Triton) based on queue depth, with scale-to-zero for cost efficiency
4.Cold start tradeoffs when scaling GPU workloads to zero, and how to mitigate them
5.When K8s + KEDA is actually worth the complexity vs simpler managed alternatives

Attendees will leave with a clear mental model of how to think about autoscaling ML workloads on Kubernetes, the tradeoffs involved, and practical patterns they can apply regardless of their ML serving stack.