Operating Elasticsearch in Kubernetes

Operating stateful applications is a challenge and Elasticsearch clusters are no exception. At Zalando, the search department has a long-term history of operating large-scale Elasticsearch clusters consisting of 100s of nodes.

In this talk Mikkel and Oliver present how Zalando utilizes Kubernetes and the operator pattern as introduced by CoreOS, to reliably and cost efficiently operate large-scale Elasticsearch clusters.

The talk will walk through how the Elasticsearch operator was designed, what problems it solves and how building it from the bottom up allowed getting it in production fast, gather more learnings and later extending the featureset to make it less manual to operate and reducing the cost of the overall infrastructure by introducing auto-scaling.

While the talk revolves around running Elasticsearch, the ideas presented also apply to running other types of stateful applications in Kubernetes. Additionally, the Elasticsearch operator presented, can be used by anyone. It is planned to be published as open-source solution in the future.