Shuo Chen
databricks, senior software engineer
Actions
Shuo Chen is a software engineer at Databricks platform team, whose domain area is focusing on high performance computing infrastructure. With the industrial experience of building the underlying network and disk infrastructure for multiple cloud provider companies, Shuo is currently focusing on building high performance multi-tenancy k8s infrastructure to support Databricks high scalable workloads.
Links
Auto-Expandable and Zero-Downtime Storage using Kubernetes CSI
At Databricks, our data storage layer uses both local disks and remote block devices.To provide a customer experience of unlimited storage, the storage solution needs to be able to automatically expand without causing downtime. However, not all cloud providers offer an infinitely expandable storage solution that is application-agnostic. To address this challenge, we have leveraged the Kubernetes Container Storage Interface (CSI) to create a hybrid mode solution, combining local and remote block devices with a self-reconciling mechanism to detect file system shortages and enable automatic expansion.
We will share our experiences in developing such a storage solution and some practices for optimizing storage layer performance and achieving cost-efficiency. We will discuss key considerations and challenges encountered during implementation and offer insights into expanding our CSI solution to various use cases, such as multi-level ephemeral storage.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top