Speaker

Will LaForest

Will LaForest

Field CTO, Confluent

Vienna, Virginia, United States

In his current position, Mr. LaForest works with customers across a broad spectrum of industries and government enabling them to realize the benefits of a data in motion and event driven architecture. He is passionate about innovation in data technology and has spent 26 years helping customers wrangle data at massive scale. His technical career spans diverse areas from software engineering, NoSQL, data science, cloud computing, machine learning, and building statistical visualization software but began with code slinging at DARPA as a teenager. Mr. LaForest holds degrees in mathematics and physics from the University of Virginia.

Area of Expertise

  • Information & Communications Technology

Topics

  • Kafka
  • Databases
  • data mesh
  • All things data
  • confluent
  • cyber security
  • stream processing
  • Data Streaming

Building a Dynamic Rules Engine with Kafka Streams

The benefit of real-time data can be measured by how frequently the data in question changes, nowhere is this more apparent than threat detection. Responding to an ever changing landscape of attacks and exploits requires a system that can not only handle the scale and dynamic nature of the data but also a dynamically changing set of detection rules. We developed Confluent SIGMA, an open source project built on Kafka Streams for the open SIGMA DSL, to handle real-time rule additions and modifications. In this talk we will cover:

* The architecture of our Kafka Streams layer that makes it possible to use external data feeds as rule input
* How we handle dynamic criteria for joins and filters
* Best practices for writing dynamic rule engines in Kafka Streams
* Upcoming improvements to Kafka Streams to support versioned rules

Although Confluent SIGMA focuses on cyber threat detection this same pattern can also be applied to any DSL (domain specific language) that would benefit from real-time stream processing. After attending you will have the framework to drive dynamic rules through Kafka Streams for any use case that might require it.

Where in the world is Franz Kafka?

Apache Kafka is the de-facto standard for event streaming and creating data pipelines that can feed a variety of different tools. It is very common for the data to have geospatial characteristics but to date there has been relatively little work done around how to leverage this natively in Kafka. The common approach is to just dump all the data into some geospatial store or toolset and do retrospective analysis and queries. This of course loses all the advantages of handling it in realtime before it ever goes to an external tool. In this talk I will discuss the creation and demonstrate the usage of geospatial UDFs in ksqlDB. I will also talk through the advantages of doing geospatial processing directly in Apache Kafka.

An Odyssey to ScyllaDB and Apache Kafka

At Confluent we focus on helping our customers move to an event driven data in motion architecture. This is not something achieved in a vacuum but highly interwoven with an ecosystem of data technologies that fit the problems our customers are solving. The variety of data technologies in active use today are astonishing to those of us who grew up in the RDBMS hammer days. Helping our customers make this transition means touching and working with all of these. In my own database odyssey, ScyllaDB has shown itself to be particularly well suited for the modern event driven architecture. In this presentation, I will cover why ScyllaDB is a good fit for people using Apache Kafka in event driven architectures, review customer examples, and discuss my usage of ScyllaDB in a high velocity data sharing effort.

Will LaForest

Field CTO, Confluent

Vienna, Virginia, United States