Your local LLM setup works. Then you push to staging and it breaks, because your app was pointing at localhost. So you switch to a cloud API, pay for tokens you didn't need, and send data you didn't intend to share.

We hit this building QCoda, an AI orchestration platform. Running agents at scale, the local-to-staging gap was a blocker. So we built LLMesh: an open-source inference mesh where compute nodes (laptops, GPU boxes, anything running Ollama) connect outward to a central hub. Your app hits one endpoint. Config never changes between environments. The mesh self-assembles from hardware your team already owns.

This talk covers what we needed, why nothing fit, the architectural flip that solved it (nodes reach the hub, not the reverse), and why we open-sourced it. I'll walk through pooling a small team's GPU capacity with no DevOps hire, no cloud contract, and zero third-party API calls. And tease the kind of LLM orchestration it unlocks.

Takeaways:

- The connection-direction pattern that kills the local-to-staging config gap
- Pooling existing hardware into a distributed inference mesh behind one endpoint
- Why open source, and what that decision actually costs
- Built-in token analytics for making real local-vs-cloud decisions
- Going from single-dev setup to team-wide mesh without a cloud API
- What you can do with a mesh that provides truly private small and large models available to your whole team

Notes for reviewers

LLMesh is fully open source (MIT, github.com/qcoda-ai/llmesh). No sales pitch. QCoda is origin-story context only. All tech discussed is open source. Possible to live demo, can discuss.