At Databricks, our data storage layer uses both local disks and remote block devices.To provide a customer experience of unlimited storage, the storage solution needs to be able to automatically expand without causing downtime. However, not all cloud providers offer an infinitely expandable storage solution that is application-agnostic. To address this challenge, we have leveraged the Kubernetes Container Storage Interface (CSI) to create a hybrid mode solution, combining local and remote block devices with a self-reconciling mechanism to detect file system shortages and enable automatic expansion.

We will share our experiences in developing such a storage solution and some practices for optimizing storage layer performance and achieving cost-efficiency. We will discuss key considerations and challenges encountered during implementation and offer insights into expanding our CSI solution to various use cases, such as multi-level ephemeral storage.