Quantization has a credibility problem. Most tutorials show you how to shrink a model, then stop right before checking whether it still works. We've all seen the "4-bit with no quality
loss!" claims that fall apart the second you run a real eval.
We've been working with LLM Compressor, vLLM's compression library, and we want to share the honest picture. Which formats hold up: FP8 is the safe bet, and NVFP4 gets a 70B model onto
one GPU if you understand the tradeoffs. Which algorithms actually earn their place: GPTQ, AWQ, and newer microscaling formats like MXFP4 for Blackwell. And where accuracy-aware
transforms like SpinQuant and QuIP-style rotations let you push to 4-bit without falling off a cliff.
Then we get to the part most talks skip: benchmarking. Not just perplexity, but HumanEval pass rates, real inference throughput on H100 and H200, and the gap between "works on paper"
and "works under load." We'll show where FP8 KV-cache quantization genuinely helps long-context workloads, and where it quietly makes them worse.
You'll leave with a workflow you can take back to your team: compress, evaluate, deploy, and know exactly what you gave up.

Audience:
ML engineers, model owners, and anyone who's been asked to "make the model cheaper" without making it worse.