Session
The Payment That Arrived on Thursday: Fixing Training-Serving Skew from Late Events
A customer pays on Monday. The ACH transfer lands on Thursday. On Tuesday, a risk model scores them as someone who hasn’t paid. Months later the model is retrained with hindsight, sees the payment on time, and learns from a world it never saw in production.
Late data isn’t an edge case in financial pipelines; it’s how money moves. Payments clear in one to three days and credit bureau data can take a month. In this session I compare how four common architectures handle it: Lambda, Kappa, a feature store and a classic three-tier medallion. Using an open benchmark of 50,000 synthetic accounts, I show why designs that recover late events create training-serving skew, and why designs that avoid skew drop the late events.
Then I walk through a fix that gets both: train on the exact feature values that were served, correct late events by writing new versions instead of overwriting, and give each event class its own watermark. In the benchmark this removed skew entirely and recovered 100% of late events. I’ll also be honest about what it doesn’t buy you: the accuracy gain was small, and the real wins are audit replay and one less failure mode.
You’ll leave with a checklist for spotting skew in your own pipelines and open-source code to test your architecture against.
Akshay Sharma
Lead Member of Technical Staff, Salesforce | Data & AI Platforms | IEEE Senior Member
Chicago, Illinois, United States
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top