A Record’s ACID trip through Kafka

In this talk I will be exploring how to use Apache Kafka’s in-built transaction API to build transactional microservices, allowing distributed systems to meet processing guarantees.

If there's one thing we know about software, it is that eventually something will fail, and you many lose data.
This is not the end. Designing for failure has brought us many useful innovations like the TCP protocol, the Erlang programming language, and even Apache Kafka itself.
It's so important, that databases have enshrined these properties as ACID compliance.

But what happens when there is more than one system in your transaction? Classically microservices have to do more than just commit changes to one database or one Kafka topic. In addition, how do you maintain exactly once processing guarantees when you may have to deal with systems failing?

Sometimes it's more than just a two phase commit, and when you are dealing with payment systems, being able to act upon a stream continuously while maintaining ACID characteristics and exactly once semantics is mandatory.

Follow along as we see what happens when systems start to degenerate, and what we can and can’t trust at scale. Learn about when to use Kafka as the transaction controller, what can and can’t be stateless, and what are the tradeoffs.

We will explore completion criteria, the routing slip pattern, the outbox pattern, and others as we go on a trip through the various methods of ensuring ACID (atomicity, consistency, isolation, and durability) compliance and exactly once processing, in an asynchronous distributed system. Leave with a few extra tools and patterns about how to make large scale systems reliable