Inference optimization is a grind. The big wins like quantization affect model quality, triggering numerous, expensive quality evals. Both host and accelerator performance improvement is a game of inches. And the engines have more flags than an Erasmus alumni meetup in Brussels.

In this talk, I will present the playbook we've developed at Modal for optimizing inference deployments, like our Kimi-K3 Shared Endpoint (available via OpenRouter) and application-specific endpoints with and for customers like Decagon.

We focus in particular on speculative decoding, which uses inference to speed up inference. I'll explain the core intuition, the latest techniques, and the relevance of the bitter lesson, with reference to our work on state-of-the-art DFlash speculators.