Distributed inference brings competing placement needs: workers benefit from fast interconnects, prefill and decode exchange KV cache, and independent serving instances need fault isolation. How can these requirements hold as a service scales and changes?
This session follows an inference service from workload definition to placement with Kthena and Volcano. We show how Kthena models serving groups and roles, and how Volcano combines hierarchical gang scheduling with multilevel network topology to place cooperating components. Concrete scenarios explore locality within worker groups, isolation across serving instances, and evolving support for group and subgroup affinity and anti-affinity. We also examine how scaling and rolling updates interact with these constraints.
Attendees will learn to translate inference architectures into scheduling policies and reason about trade-offs between communication locality, service availability, and capacity for growth.