Writing a Kubernetes operator looks easy with Kubebuilder and controller-runtime, until production traffic hits. Then the real problems begin.

This session dives into the hard parts of building production-grade operators, focusing on the reconciliation loop and managing stateful workloads safely.

Operators inherently "live in the past" because they read from a cache populated by watch events. This stale view can cause subtle race conditions, over-creation bugs, and inconsistent state. We’ll explore how to handle this safely using optimistic concurrency, resource versions, and the memory expectations pattern used by core Kubernetes controllers.

I’ll also cover designing idempotent reconciliations, deterministic resource naming, spec-hash comparisons instead of brittle DeepEqual checks, safe pod template customization, and the realities of StatefulSets and persistent volumes, including update strategies like OnDelete and volume binding pitfalls.

If you’re building or operating controllers in real clusters, this talk will help you avoid painful production mistakes.