Sourcing Operational Data in a Data Mesh with Kafka, Striim and Schema Registry

Data Mesh – a popular data management concept coined by Zhamak Dehghani – suggests patterns to work with collaborating operational systems. However, there's a few key challenges that must be addressed to ensure consumers are subscribing to domain-relevant data products rather than superfluous technical data in the form of logical change records. In this talk we will learn about how Data Mesh patterns recommend sourcing data from collaborating operational systems (OLTP databases, sensors, API data) and transforming it into Data Products in the format of actionable business data with strong SLAs/SLOs. We will talk about how to leverage Apache Kafka, Striim, and Confluent Schema registry to manage decoupled data delivery between N-number of sources to M-number of consumers, schema contracts, schema evolution, and fault tolerant data processing.