Introduction
The Deep Learning (DL) landscape is currently characterized by rapid innovation across models, frameworks, and hardware accelerators. However, this "Wild West" environment has led to a reproducibility crisis. Benchmarking new innovations often relies on ad-hoc, environment-specific scripts, making "apple-to-apple" comparisons nearly impossible. Researchers struggle to determine if performance gains stem from model architecture, specific software versions (e.g., CUDA), or hardware configurations. MLModelScope is introduced as an open-source, distributed platform designed to standardize DL benchmarking, providing a scalable and reproducible way to evaluate hundreds of models across diverse stacks.
System Architecture and Methods
MLModelScope is built on a distributed architecture designed to achieve ten core objectives, including hardware agnosticism and scalability. The platform decouples model logic from the underlying software stack through a manifest-based specification.
Key components include:
- Manifest-Based Specification: Users define evaluations using YAML-based "Model" and "Framework" manifests, ensuring environment independence.
- Distributed Agents: Lightweight agents run on target systems (cloud or edge), registering hardware capabilities (CPU, GPU, FPGA) to a central registry and fetching model assets as needed.
- Standardized Predictor API: To minimize overhead, MLModelScope binds directly to the C APIs of various frameworks.
- Multi-Level Tracing: The platform captures performance data at the model level (layer latency), framework level (operator execution), and system level (hardware counters and memory bandwidth).
Experimental Methodology
The platform was evaluated using a comprehensive suite of 37 pre-trained TensorFlow models from MLPerf, AI-Matrix, and TensorFlow Slim. These models were tested against the ImageNet validation dataset for accuracy and performance. The evaluation spanned four distinct hardware configurations: AWS P3 (Volta V100), AWS G3 (Maxwell M60), AWS P2 (Kepler K80), and an IBM P8 system (Pascal P100).
Key Insights: The "Python Tax"
A significant finding during the implementation of MLModelScope was the impact of high-level scripting languages on benchmarking accuracy. The authors identified a "Python Tax," where the use of native Python structures significantly degrades performance.
As shown in the figure, using native Python lists can make inference up to 64% slower on CPUs compared to C-based execution. Even NumPy introduces a 10-15% overhead. This underscores the necessity of binding directly to C APIs to obtain "true" hardware performance metrics.
Results and Discussion
MLModelScope's "zoom-in" tracing capability revealed critical performance bottlenecks. For instance, in a "cold-start" scenario, the interconnect (e.g., PCIe vs. NVLink) often becomes a greater bottleneck than raw GPU compute. The IBM P8 system, utilizing NVLink (33 GB/s), outperformed the AWS P3 system using PCIe3 (12 GB/s) in these scenarios, specifically reducing latency in memory-bound layers like AlexNet's fc6.
Furthermore, the platform demonstrated that model size does not always correlate with accuracy; several smaller models achieved superior accuracy-to-latency ratios. In terms of cost-efficiency, while the V100 GPU provided the lowest absolute latency for ResNet-50, the M60 GPU proved more economical based on cloud pricing models.
Conclusion and Future Work
MLModelScope successfully demonstrates a scalable approach to managing the combinatorial explosion of DL benchmarking. Future work aims to extend the platform's capabilities to include distributed training benchmarks and to expand the library of supported hardware, particularly for FPGAs. The authors are also developing a public instance of the platform to foster community-driven model comparison and curation.
Summary updated