Harshul Jain
Audible, Senior Software Engineer
Newark, New Jersey, United States
Actions
I'm Harshul, Senior Software Engineer at Audible. I build distributed systems and GenAI platforms at scale - feature stores that handle 100K transactions per second, real-time streaming pipelines, and AI Search serving 10 million users. Outside work, I've reviewed 5 books for Manning, mentored over 300 engineers on Codementor and Topmate, and built AgentShip, an open-source framework agnostic solution for writing and deploying AI agents to production.
Area of Expertise
Topics
Securing the Agentic Tool Layer: A Runtime Defense Framework for MCP Agent Deployments
Most LLM safety work guards the user prompt. MCP agents face a different problem: the attack arrives through the tools the agent already trusts. A poisoned tool description, a hidden instruction in a tool response, or a malicious server pulled from a public registry can hijack an agent without ever touching the user.
This session walks through a threat taxonomy built from 80+ techniques catalogued under SAFE-MCP, a Linux Foundation/OpenSSF project, then shows a runtime proxy that validates tool calls and responses inline, with no changes to the agent or server. In red-team testing across five model backends, it cut tool-poisoning success from 74% to under 9% and indirect injection from 47% to under 6%, adding under 120ms per call.
Attendees leave with a concrete defense architecture, the false-positive tradeoffs that matter in production, and what to vet before approving an MCP server for their stack.
LLM Inference on Edge Kubernetes: Patterns for Offline, Constrained, and Heterogeneous Devices
Edge LLM inference isn’t “cloud inference, but smaller.” Edge deployments must handle intermittent connectivity, tight memory/compute budgets, and heterogeneous accelerators (CPU-only, iGPU, small GPUs, NPUs). Teams that reuse cloud assumptions (always-on control plane, uniform hardware, centralized logging) hit failure modes that look like randomness: cold starts that take minutes, unpredictable latency, silent model corruption, and observability gaps when devices go offline. This talk presents Kubernetes-native patterns for shipping and operating LLM inference on edge clusters (single-node and small multi-node): model artifact packaging and rollouts (versioning, integrity checks, rollback safety), scheduling/isolation across heterogeneous nodes, offline-first operations (local queues + eventual sync), and observability that still works with delayed export (local buffering + minimal health signals).
Ref Repo: https://github.com/harshuljain13/llm-inference-at-scale
2 hr deep dive on LLM Inference at Scale
Most engineers using LLMs can call an API. Far fewer can explain why their model is slow, why it's running out of memory, or how the inference engines powering every major LLM API actually work.
This workshop walks through the full inference stack — from how a transformer generates a single token to serving billions of tokens a day with vLLM, SGLang, TensorRT-LLM, Ray, and KServe/llm-d. 60% explanation with live demos, 40% hands-on exercises. Attendees leave with a running vLLM server they benchmarked themselves.
Based on the open-source practitioners handbook being built live at github.com/harshuljain13/llm-inference-at-scale
Beyond Frameworks: Building Runtime-Agnostic AI Agents
AI agent frameworks are proliferating - Google ADK, LangGraph, OpenAI Agents SDK, CrewAI - each with distinct paradigms, tool interfaces, and deployment models. Meanwhile, tool integration standards like MCP (Model Context Protocol), memory solutions like mem0 and Supermemory, and observability platforms like LangFuse and Opik are emerging to solve interoperability, but adoption remains fragmented. Teams invest months building agents on one framework with tightly coupled memory, observability, and tooling systems, only to discover they need capabilities from another. The result? Rewrites, lock-in, and production delays.
This talk introduces a different approach: runtime-agnostic agent architecture. Instead of coupling agent logic to a specific framework, we separate what the agent does from how it executes. Write your agent once, run it on any engine and integrate tools and memory through standardized abstractions.
Drawing from AgentShip an open-source production layer for AI agents, we'll explore:
- **The Pluggable Engine Pattern**: An `AgentEngine` abstraction that enables swapping ADK for LangGraph (or future runtimes) without touching agent code
- **MCP Integration Layer**: Standardized tool discovery and invocation using Model Context Protocol, making tools portable across frameworks
- **Pluggable Memory Systems**: Abstracted short-term and long-term memory with support for custom implementations, mem0, Supermemory, or simple in-memory stores
- **Pluggable Observability**: Framework-agnostic tracing and monitoring with support for LangFuse, Opik, or custom observability backends
- **Declarative Agent Configuration**: YAML-driven definitions that eliminate framework-specific boilerplate
- **Unified Streaming & Sessions**: Consistent behavior across engines with three streaming modes and pluggable session storage
- **Agent-to-Agent Orchestration**: Hierarchical multi-agent workflows where sub-agents become tools
We'll demonstrate these patterns through an health AI assistant where a primary agent orchestrates five specialized sub-agents for conversation analysis, medical reports, and action tracking all running on interchangeable backends.
Attendees will leave with architectural patterns they can apply immediately, whether building new agent systems or refactoring existing ones away from framework lock-in.
CNCF-Hosted Co-located Events North America 2026 Sessionize Event Upcoming
AI Engineer World's Fair 2026 Sessionize Event
MCP Conference New York Sessionize Event
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top