The Design and Implementation of a Scalable Deep Learning Benchmarking Platform

Cheng Li, Abdul Dakkak, Jinjun Xiong, Wen-Mei W. Hwu

Introduction

The Deep Learning (DL) landscape is currently characterized by rapid innovation across models, frameworks, and hardware accelerators. However, this "Wild West" environment has led to a reproducibility crisis. Benchmarking new innovations often relies on ad-hoc, environment-specific scripts, making "apple-to-apple" comparisons nearly impossible. Researchers struggle to determine if performance gains stem from model architecture, specific software versions (e.g., CUDA), or hardware configurations. MLModelScope is introduced as an open-source, distributed platform designed to standardize DL benchmarking, providing a scalable and reproducible way to evaluate hundreds of models across diverse stacks.

System Architecture and Methods

MLModelScope is built on a distributed architecture designed to achieve ten core objectives, including hardware agnosticism and scalability. The platform decouples model logic from the underlying software stack through a manifest-based specification.

MLModelScope System Architecture: The platform uses distributed agents and a centralized registry to manage evaluations across diverse hardware and software environments.

Key components include:

  • Manifest-Based Specification: Users define evaluations using YAML-based "Model" and "Framework" manifests, ensuring environment independence.
  • Distributed Agents: Lightweight agents run on target systems (cloud or edge), registering hardware capabilities (CPU, GPU, FPGA) to a central registry and fetching model assets as needed.
  • Standardized Predictor API: To minimize overhead, MLModelScope binds directly to the C APIs of various frameworks.
  • Multi-Level Tracing: The platform captures performance data at the model level (layer latency), framework level (operator execution), and system level (hardware counters and memory bandwidth).

Experimental Methodology

The platform was evaluated using a comprehensive suite of 37 pre-trained TensorFlow models from MLPerf, AI-Matrix, and TensorFlow Slim. These models were tested against the ImageNet validation dataset for accuracy and performance. The evaluation spanned four distinct hardware configurations: AWS P3 (Volta V100), AWS G3 (Maxwell M60), AWS P2 (Kepler K80), and an IBM P8 system (Pascal P100).

Key Insights: The "Python Tax"

A significant finding during the implementation of MLModelScope was the impact of high-level scripting languages on benchmarking accuracy. The authors identified a "Python Tax," where the use of native Python structures significantly degrades performance.

Normalized Latency Comparison: This figure illustrates the significant performance overhead (the "Python Tax") incurred when using Python or NumPy compared to native C-based execution across different batch sizes.

As shown in the figure, using native Python lists can make inference up to 64% slower on CPUs compared to C-based execution. Even NumPy introduces a 10-15% overhead. This underscores the necessity of binding directly to C APIs to obtain "true" hardware performance metrics.

Results and Discussion

MLModelScope's "zoom-in" tracing capability revealed critical performance bottlenecks. For instance, in a "cold-start" scenario, the interconnect (e.g., PCIe vs. NVLink) often becomes a greater bottleneck than raw GPU compute. The IBM P8 system, utilizing NVLink (33 GB/s), outperformed the AWS P3 system using PCIe3 (12 GB/s) in these scenarios, specifically reducing latency in memory-bound layers like AlexNet's fc6.

Furthermore, the platform demonstrated that model size does not always correlate with accuracy; several smaller models achieved superior accuracy-to-latency ratios. In terms of cost-efficiency, while the V100 GPU provided the lowest absolute latency for ResNet-50, the M60 GPU proved more economical based on cloud pricing models.

Conclusion and Future Work

MLModelScope successfully demonstrates a scalable approach to managing the combinatorial explosion of DL benchmarking. Future work aims to extend the platform's capabilities to include distributed training benchmarks and to expand the library of supported hardware, particularly for FPGAs. The authors are also developing a public instance of the platform to foster community-driven model comparison and curation.

Summary updated

Abstract

The current Deep Learning (DL) landscape is fast-paced and is rife with non-uniform models, hardware/software (HW/SW) stacks. Currently, there is no DL benchmarking platform to facilitate the evaluation and comparison of DL innovations, be it models, frameworks, libraries, or hardware. As a result, the current practice of evaluating the benefits of proposed DL innovations is both arduous and error-prone - stifling the adoption of the innovations. In this work, we first identify 10 design features that are desirable within a DL benchmarking platform. These features include: performing the evaluation in a consistent, reproducible, and scalable manner, being framework and hardware agnostic, supporting real-world benchmarking workloads, providing in-depth model execution inspection across the HW/SW stack levels, etc. We then propose MLModelScope, a DL benchmarking platform that realizes these 10 design objectives. MLModelScope introduces a specification to define DL model evaluations and provides a runtime to provision the evaluation workflow using the user-specified HW/SW stack. MLModelScope defines abstractions for frameworks and supports the board range of DL models and evaluation scenarios. We implement MLModelScope as an open-source project with support for all major frameworks and hardware architectures. Through MLModelScope's evaluation and automated analysis workflows, we perform a case-study analysis of 37 models across 4 systems and show how model, hardware, and framework selection affects model accuracy and performance under different benchmarking scenarios. We further demonstrate how MLModelScope's tracing capability gives a holistic view of model execution and helps pinpoint bottlenecks.
BibTeX
@inproceedings{li2020design,
  author = {Cheng Li and Abdul Dakkak and Jinjun Xiong and Wen-Mei W. Hwu},
  title = {{The Design and Implementation of a Scalable Deep Learning Benchmarking Platform}},
  booktitle = {CLOUD},
  year = {2020},
  doi = {10.1109/cloud49709.2020.00063},
}
Figure