Ford Motor Company, a leading global automotive manufacturer, maintains a significant operational presence worldwide. Within Ford, the Data Operations (DataOps) team for Advanced Driver Assistance Systems (ADAS) is entrusted with the critical role of processing petabyte-scale datasets derived from lidar, radar, and video sources. The data processed by the team is instrumental in both enhancing existing ADAS features and facilitating the development of future advancements in the realms of ML/AI.

A pivotal aspect of managing data at such a scale involves the meticulous optimization
of tasks within a High-Performance Compute (HPC) cluster to achieve optimal parallelization and efficient queuing mechanisms. To do this, the ELK stack is essential to continuously predict and adjust the size of tasks for precise calibration of our computational resources. In this talk, we will share our journey, focusing on the challenges encountered in scaling our operations and the architectural solutions we implemented to facilitate adaptive task sizing using log analysis.