Tired of cloud API latency, rate limits, and per-token bills? Open-weight models have gotten good enough, and runtimes fast enough, that your workstation can do real work. This session covers how to run, quantize, and serve open-weight models on your own hardware, and where the limits are.
We'll start with what a model actually is (weights, parameters, tokens) and why running it locally makes sense (or doesn't). Then we'll get practical:
Quantization: choosing a method and bit width for your memory budget, and what you give up
Inference engines: picking and configuring the right runtime for your hardware and use case
Hardware sizing: rules of thumb for memory bandwidth, VRAM, and context window management on local silicon
You'll leave knowing what your machine can realistically run, how fast, and when the cloud is still the better call. Bring your questions.