In large-scale AI inference clusters, multi-tenant workloads require both efficient GPU utilization and dynamic RDMA networking. However, heterogeneous GPU interconnect technologies inevitably lead to multi-level network topologies, such as scale-up networks and RDMA spine–leaf structures.
These diverse topologies introduce several challenges: Dynamic topology discovery and health detection across multiple layers, including scale-up, RDMA spine, and RDMA leaf. Second, Topology-aware scheduling that supports priority-based placement and ensures GPUs leverage optimal communication paths.Third, Validation at scale, requiring cost-effective simulation of large, multi-level topologies instead of relying on expensive hardware.
In this talk, it will share practical approach of topology discovery to help Kueue to achieve topology-aware scheduling, and showcase how Kwok simulates thousands of virtual nodes with multi-level topologies, enabling large-scale validation at zero hardware cost.