You have a cluster. You can scale up and down at a moment's notice. You can roll updates without taking down your services. So are all your problems solved?

Not quite. You still have the issue of keeping your microservices in sync.

You want each service to be able to make its own decisions autonomously. They therefore need to store their own copy of the data that they need. But that copy is not always up-to-date. To achieve consistency at scale, you need to design messages that will let pods converge within a reasonable time. Those messages need to be idempotent so that repeats do not cause duplication. And they need to be commutative so that pods will converge even after receiving messages in different orders.

These are not trivial properties. They are difficult to achieve in a traditional mutation-based paradigm. But if your model was immutable, then you would get these properties almost for free. Every copy of an immutable record is just as good as any other, and it will never be out-of-date. The trick then is how to model your application using only immutable records.

Let me show you how.

We'll begin by analyzing your problem domain using a causal historical model. You will see how workflow patterns take us beyond state-machine thinking and solve technical problems and business problems simultaneously. Then we will map that immutable model into messages so that we can deploy it to our cluster. Use AMQP, Kafka, or any messaging infrastructure that you are comfortable with. Finally, we will employ the sidecar pattern to translate the immutable model back into the mutation-based entities that our application containers expect.

Kubernetes is a wonderful tool for managing workloads. Now that you have it, apply the patterns of immutable architecture to manage data within your distributed cluster.