Managing AI/ML workloads with GPUs on Kubernetes presents formidable challenges due to complex job management and scheduling, along with the need for substantial specialized computing resources, such as GPUs, which are not readily available.

This talk introduces Knavigator, an open-source framework and toolkit designed to support developers of Kubernetes systems. Knavigator facilitates the development, testing, troubleshooting, benchmarking, chaos engineering, performance analysis, and optimization of AI/ML control planes with GPUs in Kubernetes.

Knavigator enables tests on Kubernetes clusters using both real and virtual GPU nodes, allowing for large-scale testing with limited resources, such as a laptop.

Through real examples and demos, this presentation will showcase Knavigator's capabilities in feature validation, performance, load testing, and reliability testing. It will also highlight how Knavigator enhances the fault tolerance of large model training jobs in Kubernetes.