As autonomous driving shifts to end-to-end models, Horizon Robotics faces unprecedented scale. Thousands of concurrent, sub-minute workflows pushed infrastructure to its limits, where traditional orchestration and "Informer Choke" caused severe instability and resource waste.
We re-architected our platform to handle 8 million daily tasks, saving 1,000+ CPU nodes. We will dive into:
1. Orchestration Re-engineering: Replacing per-workflow agents with a centralized Argo Workflows Agent Service. This "Agentless" approach cut infrastructure costs by 2.4M CNY/month and saved 1,000+ CPU nodes.
2. Scheduling & State Optimization: Overcoming Informer bottlenecks by refactoring metadata, reducing VolcanoJob size by 30%, and slashing sync latency from 4 minutes to <10 seconds.
3. Stability & Multi-Tenancy: Leveraging Kueue and Volcano to implement elastic quotas and resolve preemption conflicts, ensuring system stability in large-scale multi-tenant environments.