Daan Gerits
Founder @ Shono
Aarschot, Belgium
Actions
Daan has been at the forefront of the data landscape since the early days of the data (r)evolution designing and implementing data-driven architectures for startups, telcos, airports, and banks. While the early focus was on the usual big data suspects, the focus has shifted in the last decade toward data streaming and event-driven systems. The result? A vision that goes well beyond your typical Kafka use-case into fundamentally changing how businesses work today.
As the Chief Data Officer at KOR Financial, a new institutional financial transaction data management and processing platform, Daan designed and implemented a bold event-driven vision at Petascale. KOR Financial proved to be the ideal platform for building out the next-generation business, one that is driven by individual events instead of their results.
Recently Daan founded a new startup called Shono.io with a mission to turn the integration nightmare into a data opportunity through an event-driven approach.
Links
Topics
Embracing the event-driven mindset
While Apache Kafka works great for just shipping messages from point A to B, there is much more we can do with it. What would happen if we were to build applications on top of Kafka using the event-driven mindset? What would an event actually look like? And where does state enter the picture?
Let’s introduce the event-driven mindset:
Reality is event-driven by nature
- Events cause events
- Events are immutable
- State is a consequence
- Time is relative
This talk takes a practical approach to the event-driven mindset, talking about the consequences of how to model your concepts, decide on events and the topics they go on, and leverage state stores to achieve your goals. By the end of the session, you’ll understand what an event driven system is, how it differs from more traditional approaches, and how to use this mindset as part of your toolbox to design and implement your next application!
The Possibilities And Pitfalls of Writing Your Own State Stores
The Possibilities And Pitfalls of Writing Your Own State Stores
Building an event-driven system will inevitably lead you to exposing your data through APIs to make your data accessible to non-streaming solutions. At first glance, Kafka Streams provides statestores which we could use to build our APIs directly onto Kafka. But all store implementations are key/value based which is fine when you only retrieve information by key. API’s however require a bit more “searchability”.
Writing your own state store is certainly possible, but it is challenging. At KOR, we went through this process and implemented a state store on top of Nitrite Database, an embedded document database. This allows you not only to retrieve your documents using a key, but also to search through the values in the store using a MongoDB-like API. On the surface, state stores seem straightforward, but the devil is certainly in the details. How does partitioning fit into this story and how do you make sure everything keeps running smooth, even after restarting or scaling your applications?
We made the project open source for everyone out there wanting to try this approach, but most of all we want to tell you about the dragons we encountered. Join me in a journey of ups and downs that starts with a simple requirement (host an API), through implementing a custom state store and finishes off by describing the challenges we encountered getting our APIs deployed. Don’t expect all “roses and sunshine”. While hosting APIs on Kafka is possible, there are some consequences that we just couldn’t overcome … yet.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top