I want to open by showing you something, not telling you about it. I put a proxy in front of a coding assistant I use every day and looked at what it actually sends. Repo structure, file paths, the contents of files I didn't have open, and the full system prompt it assembled before it asked the model anything. None of that is a breach, and none of it is undocumented. It is the product working as designed, and I had agreed to all of it. I had just never looked.

Then I turn the Wi-Fi off, and it stays off for the rest of the session.

What interests me is not that you can run a model locally. Most of us tried a 7B and came away unimpressed. What has changed is that a 120B mixture-of-experts model now runs on a device that fits in a jacket pocket and draws 65 watts, and you can put that device behind the same endpoint as a 128GB AMD Ryzen AI Max+ 395 laptop and treat the pair as one piece of infrastructure.

So, this is not a hardware session. It is about what you build once you have it.

Foundry Local handles the awkward part: the machines have completely different accelerators, and I would rather not write device plumbing for each one. A LiteLLM gateway sits in front of them, so everything upstream sees a single OpenAI-compatible endpoint and has no idea which box answered. Then I run a coding agent against a private repo and let the work spread across the fabric. Classification, redaction and embeddings land on the small device, because latency is what it is good at. Planning and code generation go to the 120B. A local Langfuse instance shows the trace filling in as it happens, so you can see which machine served which span, and the network monitor sits at zero throughout.

Memory is the part I think we under-examine. A prompt leaks once, but an agent's memory is the distilled version of everything you have ever worked on, and most of it lives in a vector store nobody can read. I run a local memory layer so the agent keeps context across sessions and repositories, and the memory itself is inspectable — plain files you can open, with provenance attached, rather than a black box you have to trust.

I finish with the parts that do not work. Long-context prefill is slow on this hardware, and I am not editing the pause out. Quantisation costs you something real, and I will show you an eval rather than assert it. The reasoning gap against frontier models is genuine. Local is also not the same as compliant, which is a distinction I spend much of my working life on. I end with a routing pattern that decides what stays local based on sensitivity and point to Foundry Local on Azure Local as the supported path for running this across a cluster rather than a bench.

What you will see running live

A coding agent working across a full private repo, with the work distributed across local machines and nothing leaving the room.
The trace on screen showing which machine handled which part of the job.
An agent memory layer you can actually read, built up across sessions.
A batch agent that has been running since before the session started, and the maths on what the same throughput would have cost through a frontier API.

What I hope you take away

A way to check what your own toolchain sends, which takes about twenty minutes to set up. Why the wall you hit with local models was bandwidth rather than capacity, and what that means for model choice.
A concrete architecture for routing work across heterogeneous local hardware, rather than treating local as an all-or-nothing swap.
An honest sense of where this approach stops being the right answer.

All the hardware is on a bench afterwards. Come and pick it up, see how warm the small one gets, and ask what it cost.