SPDB's big data platform serves over 300 applications in 68 sites including the headquarter, domestic and oversea branches& subsidiaries, with a scale of 2000+ nodes. The platform adopts a mixed achitecture across YARN and Kubernetes, which facilitates the cloud-native transformation while maintaining operational continuity.
STORAGE is one of the core challenges in the platform. SPDB has operated various storage solutions, including Local Volume, NAS, Longhorn, and especially Piraeus, which is a CNCF Sandbox project that SPDB has participated in. Besides, SPDB is also experimenting with Apache Ozone as a possible path to bring the data engine onto Kubernetes.
This session will share SPDB's experience in testing, evaluating and operating those storage solutions, with a focus on the performance and the best practice the team has learnt through the process. It will also introduce CSI-NFS-Export project, which SPDB is developing to help scale applications on top of a block volume.