Distributed systems promise high availability, reliability, and fault tolerance—often through elegant abstractions like Raft. But what happens when those abstractions leak?

This is a story about how managing cluster membership in our Kubernetes distribution went wrong: diverging cluster membership states across Dqlite (distributed sqlite), etcd, and clients after node failures, joins, and restarts. We will discover why these issues occur- stale state and missing guarantees that are supposed to be hidden behind a simple API.

The key takeaway for this talk is that if you work on distributed systems you need to understand the underlying mechanics to design resilient systems as distributed problems can leak from their abstraction.

I will show you at the hand of this real world example how you can use diagrams to better communicate, understand and argue about distributed system problems.