Session
"Where's My Event?": Reconciliation as Infrastructure for Event-Driven Systems
Once microservices hit real scale, event-driven architecture — built on streaming pipelines — stops being optional and becomes the only way things hold together. Teams decouple, systems absorb bursts, everyone ships independently. It works great, until an event gets stuck three services back and nobody notices.
"Where's my event?" is the question that eventually lands on someone's desk — usually the team furthest downstream, closest to the customer, with the least visibility into where things actually broke. I ran into this while building a payments platform at the end of a long, multi-team streaming pipeline, handling transaction, settlement, and payout events for a large-scale marketplace. When something upstream failed a handshake or an event stalled mid-flight, we didn't get an error — we got a support ticket and a seller asking why a payout they were expecting never showed up. And here's the part that's easy to miss: if no team ever finds that lost event, every individual team still looks fine on paper. Nobody's dashboard shows red. But the company loses anyway — real revenue, and slower-to-notice, real trust.
This talk digs into why that gap opens up in event-driven systems, and what it takes to close it — treating reconciliation as infrastructure in its own right, not an afterthought. I'll walk through the two-way reconciliation pipeline we built: one pass matching "expected" transactions from upstream against what actually landed downstream, and a reverse pass to catch cases where the reconciliation pipeline itself was quietly dropping events. That combination took detection time from days down to minutes — and gets close to seconds wherever systems can share data directly — while getting us close to zero missed transactions. I'll share the tradeoffs that mattered most: what to reconcile against, how to avoid burying the one lost event in noise, and how to measure impact in dollars and affected customers, not just event counts.
This isn't just a payments problem, or even just a niche one. The same blind spot shows up anywhere something sits downstream consuming data it didn't generate — a service reading events off a queue, or an agent reading tool outputs and context it never produced itself. Same gap. Same need for reconciliation built in as infrastructure, not bolted on after.
Ram Pedapatnam
Staff Software Engineer, Walmart
Dublin, California, United States
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top