Session
kubectl Is Lying to You: What a Live Visualizer Taught Me About Agent Substrate
Run `kubectl get pods` on an Agent Substrate cluster and you see a few quiet worker pods. Meanwhile untold numbers of agents are restoring from snapshots, queuing for slots, and checkpointing back to object storage. The interesting state isn't in pods. So I built Substrate Scope, an open-source live visualizer that shows the runtime as it really works: worker bays, a restore queue, the snapshot shelf, and agents moving between them in real time. Its buttons act on the real cluster: they scale the WorkerPool, run a demand-driven autoscaler, fire surge traffic, and reset a wedged pool.
The visualizer I built turned out to be the best way to find Substrate's edges and frankly, to explain how it works for those of us who need crayons to understand things (that would be me). On my laptop cluster I ran 18 agents on two worker pods and served over a thousand real sessions, with a minimum reserved capacity 4.5x lower than the same agents as always-on pods. This talk is a live demo plus an accounting and discussion of what broke along the way and how we work through it.
- My tool ended up with lost running sessions because I read the actors list. Every finished session leaves a Suspended actor behind; mine passed through thousands of entries in a day and started returning partial results. The workers list is the source of truth.
- CPU and memory are the wrong autoscaling signals! Workers are slot-bound and active sessions idle at single-digit mCPU. Substrate rejects when the pool is full instead of queuing, so the real demand signal is in your callers' retry loops.
- My autoscaler made things worse functionally worse. On a serial model backend, more workers just piled more requests onto one generation queue. Everything piled up and timed out and it was a hot mess. We'll look at how we address this.
- Killed sessions left ghost actors pinning workers while the pool looked idle. This made me sad but it's fixable.
- "Pods vs. substrate" charts lie if you compare usage. Idle pods use almost no CPU but reserve it forever. Reservation is the number to keep an eye on.
You'll leave knowing what to watch on a Substrate cluster, what to scale on, which failures to expect under load, and a tool you can point at your own cluster today.
Mike Moore
Solo.io · Solutions Engineer · webofmike.com
Portland, Oregon, United States
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top