Stop Paying the K/V Cache Tax: Adaptation Inside the Inference Loop

We treat a model's weights as frozen the moment training ends, so we adapt the
only way that allows — stuffing examples into the prompt and paying the K/V Cache
Tax in memory and throughput. The 2025–2026 research moves adaptation into the
inference loop instead.

I'll demo four ways to adapt a frozen LLM at inference, live on a single Mac:
three primitives from the literature — Test-Time Training (a discarded in-flight
gradient step), TAO (reward-based tuning, no labels), and δ-mem (online memory,
zero added context) — plus a fourth of my own, Reinforce-or-Retrain (RoR): a
confidence-gated loop that reinforces a confirmed answer and spends a gradient
step only when a human corrects it. Every demo runs at roughly [your measured ms]
per step, off the critical path, with nothing leaving the box. I'll close on why
the cloud's multi-tenant bottleneck makes local adaptation inevitable.

KEY TAKEAWAYS
- Frozen weights are a deployment default, not a limit — gradient, reward, and
memory each adapt a shipped model mid-session.
- Most "adaptation" in production is in-context learning in disguise; moving it
into the loop removes the memory and throughput tax.
- Reinforce-or-Retrain (RoR) — reinforce when right, retrain when wrong — is a
control loop you can lift into your own stack, shared in full.

Fine-Tuning Frontier Models on Apple Silicon: A Practical Guide to MLX-Powered Local LLMs

Abstract: As organizations scramble to deploy generative AI at scale, the economics and privacy implications of cloud-based inference are becoming untenable for many use cases. This talk presents a comprehensive, hands-on exploration of fine-tuning and deploying frontier-class language models on Apple Silicon hardware using the MLX framework — an approach that delivers production-grade performance at a fraction of cloud costs while keeping data on-device. Drawing on real-world deployment experience, we'll walk through the full pipeline: from model selection and quantization strategies, through LoRA/QLoRA fine-tuning workflows, to production-ready inference optimization. We'll cover practical benchmarks comparing MLX against traditional frameworks, discuss the trade-offs between model size, quantization level, and task performance, and share lessons learned from deploying fine-tuned models in enterprise environments. The talk will include live demonstrations of model loading, fine-tuning, and inference on M-series chips, along with actionable recommendations for teams evaluating on-device AI deployment. Whether you're an ML engineer looking to reduce inference costs, a solutions architect designing privacy-first AI systems, or a researcher exploring efficient model adaptation, this session provides the technical depth and practical guidance needed to get started with MLX-powered local AI.

Key Takeaways:
•⁠ ⁠MLX delivers competitive inference performance on Apple Silicon with dramatically lower costs than cloud alternatives
•⁠ ⁠QLoRA fine-tuning on M-series chips makes frontier model adaptation accessible without GPU clusters
•⁠ ⁠On-device AI enables privacy-preserving deployments that cloud inference simply cannot offer
•⁠ ⁠Production deployment patterns for MLX include model quantization, batch optimization, and memory management strategies
•⁠ ⁠The enterprise case for local AI hinges on cost reduction, data sovereignty, and latency improvements

Session format
Breakout sessions (45–60 minutes)

Track
Operations & Internal AI

Level
Introductory and overview

Business Function Tags
Operations

Regarding the strategic relevance of this session, particularly given its focus on local deployments rather than traditional cloud infrastructure. The presentation is designed to contrast the high cost of cloud-only LLM utilization with the efficiency of edge computing.

In particular, the session will explore how to architect federated inferencing pipelines—demonstrating how organizations can strategically divide workloads, running heavier components in the cloud while routing targeted tasks to local edge devices. Furthermore, this talk will address how to ensure local inferencing is optimized for peak performance on modern hardware, allowing attendees to walk away with a clear blueprint for building balanced, cost-effective, and highly performant hybrid AI systems.

Learning Objective 1:
Implement Cost-Effective Local Inference: Attendees will learn how to evaluate and deploy frontier-class language models on Apple Silicon hardware, enabling production-grade AI performance while significantly reducing dependence on expensive cloud-based infrastructure.

Learning Objective 2:
Master On-Device Fine-Tuning Workflows: Participants will gain hands-on knowledge of the end-to-end MLX pipeline, including model selection, quantization strategies, and LoRA/QLoRA techniques, allowing them to adapt high-performance models to specific business tasks without the need for costly GPU clusters.

Learning Objective 3:
Design Privacy-First AI Architectures: Attendees will understand how to build and maintain data sovereignty by keeping sensitive information on-device, effectively addressing security and privacy concerns while improving system latency for enterprise-grade applications.

I believe this edge-to-cloud bridge will offer a unique and highly practical perspective for cloud professionals looking to optimize their generative AI budgets and architectures.

Note: The views and research presented during this session are my own and do not necessarily reflect the views or positions of AT&T.

When Your Agent Is Wrong: Confidence-Gated Routing and Failure Handling for Multi-Server MCP

Multi-agent MCP systems make a hard question unavoidable: when an agent produces an answer by calling several tools across several servers, when do you trust it, when do you retry, and when do you escalate to a human? Most setups answer with static rules that don't improve.

I'll demonstrate a confidence-gated coordination pattern for this, built as a working reference over real MCP servers. The agent routes each result by a confidence-and-correction signal: a confident, human-confirmed answer reinforces that path; a corrected answer triggers a different one — retry, reroute, or adapt. Reinforce when right, retrain when wrong — a routing-and-failure-handling policy that sits over your MCP tool calls.

I'll walk the reference end to end on one concrete case — a customer-care agent deciding a billing credit across account, policy, and issuance servers — showing where the gate lives in the MCP topology, how state flows across servers, and how retries and human escalation are handled. I'll be honest about its limits and where it breaks.

You'll leave with a coordination pattern, and a working reference you can adapt into your own multi-server MCP architecture.