Session

Dead Reckoning for gRPC Streams: Resumable AI Inference

Server-streaming gRPC is the natural fit for LLM token delivery, until the stream drops. Today, a mid-stream failure means retrying from token zero: wasteful, expensive, and broken for users. HTTP solved this with Range headers in 1999. gRPC has no equivalent for streams.

This talk introduces a resumable streaming protocol built on top of standard gRPC. Using a session cursor, a lightweight server-side token journal, and periodic ACKs, dropped streams resume from the last confirmed token, not from scratch. We'll cover why this is a checkpointing problem (not a retry problem), how to size the journal TTL, and where LLM non-determinism makes naive resumption unsafe. Live demo: a gRPC-streamed LLM generation gets killed mid-sentence and resumes seamlessly on reconnect.

Harish Kotra

Developer Relations, Hackathons Specialist & A No-Code Educator

Hyderābād, India

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top