FZP-Index: Predictive Performance Scoring for Data Interchange Formats in AI Workloads

High-performance AI pipelines require efficient data interchange to minimize latency and maximize throughput across microservices, streaming platforms, and analytics engines. Existing evaluations of JSON, Protobuf, Parquet, or Arrow are ad-hoc and often ignore HPC-relevant factors such as memory bandwidth, vectorized execution, zero-copy data movement, and multi-core CPU scaling.

I present FZP-Index, a predictive performance scoring methodology that models encoding, decoding, transfer, and consumption costs under hardware-aware workloads. It incorporates schema features, vectorization efficiency, cache/memory behavior, and network parameters to produce a normalized score that predicts end-to-end latency and throughput on HPC-class nodes.

This approach enables adaptive format selection, guiding AI pipelines to choose the optimal representation in real-time, minimizing CPU, memory, and network bottlenecks. I evaluate FZP-Index with streaming pipelines, multi-core servers, and ML inference workloads, demonstrating significant performance gains and providing a reproducible, HPC-ready framework for high-performance AI data interchange.