Streaming data applications often need to access significant amounts of context in order to make sense of any given Kafka event. Whether such context comes from databases, third party APIs, other high rate streams, or all of the above, stateful microservices provide an efficient way to execute arbitrary application logic with memory latency access to large amounts of relevant context, at the throughput of the fastest data source.

In this talk, we’ll discuss entity-parallel stateful microservices architectures, how they’re partitioned, how they scale, and how to make them robust against failure. We’ll examine a real-world case study of combining half a dozen million event per second firehoses in real-time to autonomously monitor and analyze a nationwide network.

With a concrete use case in hand, we’ll explore patterns for implementing massively parallel time-coherent state machines that mirror the real world. We’ll look at how to continuously compute real-time aggregations and reductions. And we’ll demonstrate how to live rank high level real-time entities by biggest impact, with the intent of focussing users’—and other automated systems’—attention on what matters most right now.