Protecting Your Control Plane: A Deep Dive into kube-apiserver Memory Exhaustion

Based on real-world Red Hat support case data, this session analyzes incidents where OpenShift / Kubernetes clusters experienced unavailable kube-apiserver instances and memory-exhausted control planes. Our investigation uncovered a recurring root cause: excessive memory consumption by the kube-apiserver, driven by high-cost LIST and WATCH operations. High-cost LIST are a common issue when you retrieve many large objects from etcd. This particularly happens with secrets, jobs/cronjobs, configmaps, tekton pipeline(runs) or custom operators with "large" CRDs. In particular, we observed how a single tenant, intentionally or unintentionally, can significantly degrade or even disable the control plane simply by issuing a few aggressive LIST requests. For a blog post around this topic see https://kubernetes.io/blog/2024/12/17/kube-apiserver-api-streaming/

Attendees will learn why LIST requests are particularly problematic in Kubernetes, especially in multi-tenant environments.

The session will also examine existing Kubernetes mechanisms designed to protect the control plane, such as --max-requests-inflight, API Priority and Fairness (APF) and chunking of requests from a client side. We'll discuss how these safeguards work, where they fall short, and what administrators can do to mitigate these risks.

By the end of this session, you'll understand the hidden dangers of poorly scoped LIST calls, and gain practical strategies to secure your cluster’s control plane against memory exhaustion scenarios.

The talk will be closed with some insights in what Kubernetes 1.33 changes in terms of memory consumption of the kube-apiserver as KEP-5116: Streaming Encoding for LIST Responses (https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/5116-streaming-response-encoding) should be included in this release.