Imagine a system designed to process millions of events per day, but as use grows, data mysteriously seems to disappear, only to re-appear later. We’ll talk about the what, why and how we investigated the symptoms, overturned some assumptions and finally delivered a resilient, serverless solution

The talk explains an issue we encountered with a serverless architecture running in AWS to process logs, how we investigated the issue, before talking about how we improved the product to deal with failures more efficiently.

The issue was enventually found to be caused by ‘Eventual Consistency’, something that is a potential issue in many distributed system, but we’ve found people aren’t always aware of. We’ll also talk about why reading documentation in depth is important, not just skim-reading as at first glance, eventual consistency didn't seem to be a problem that would apply to us.