The benefit of real-time data can be measured by how frequently the data in question changes, nowhere is this more apparent than threat detection. Responding to an ever changing landscape of attacks and exploits requires a system that can not only handle the scale and dynamic nature of the data but also a dynamically changing set of detection rules. We developed Confluent SIGMA, an open source project built on Kafka Streams for the open SIGMA DSL, to handle real-time rule additions and modifications. In this talk we will cover:

* The architecture of our Kafka Streams layer that makes it possible to use external data feeds as rule input
* How we handle dynamic criteria for joins and filters
* Best practices for writing dynamic rule engines in Kafka Streams
* Upcoming improvements to Kafka Streams to support versioned rules

Although Confluent SIGMA focuses on cyber threat detection this same pattern can also be applied to any DSL (domain specific language) that would benefit from real-time stream processing. After attending you will have the framework to drive dynamic rules through Kafka Streams for any use case that might require it.