HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems

Zihan Jiang, Lei Wang, Xingwang Xiong, Wanling Gao, Chunjie Luo, Fei Tang, Chuanxin Lan, Hongxiao Li, Jianfeng ZhanarXiv 2020 · arXiv

TL;DR

This paper introduces HPC AI500 V2.0, a comprehensive framework for benchmarking High-Performance Computing (HPC) AI systems. It addresses the challenges of creating fair, repeatable, and representative benchmarks by proposing a layered system abstraction, standardized rules, and novel metrics like "Valid FLOPS" to accurately measure and rank AI system performance.

What Problem Does This Paper Address?

Benchmarking HPC AI systems is a complex challenge. Existing benchmarks often lack equivalence (making comparisons unfair), relevance (not reflecting real-world AI workloads), representativeness (covering only a narrow set of tasks), affordability (too costly to run at scale), and repeatability (results vary significantly). This makes it difficult to accurately assess, optimize, and rank the performance and energy efficiency of these powerful systems, hindering progress in the field.

Methods

The authors propose a novel methodology for HPC AI benchmarking, built around a nine-layer abstraction of an HPC AI system. This allows for testing individual layers while keeping others intact, ensuring equivalence. They define three benchmarking levels: hardware, system, and free, each with specific rules for flexibility and comparability. Key innovations include:

  • Layered System Abstraction: Breaking down the HPC AI system into nine distinct layers (from hardware to problem domain) to systematically control variables during benchmarking.
  • Benchmarking Rules: Explicit rules for each layer to ensure equivalence, repeatability, and replicability across different systems and scales.
  • HPC AI Roofline Model: A performance model adapted for HPC AI systems, emphasizing communication alongside computation, to identify bottlenecks and guide optimizations.
  • Novel Metrics: "Valid FLOPS" (VFLOPS) and "Valid FLOPS per Watt" (VFLOPS/Watt) are introduced to penalize systems that fail to achieve target quality, providing a more holistic performance evaluation.
  • Benchmark Selection: A subset of AIBench workloads (Image Classification and Extreme Weather Analytics) was chosen for their representativeness, affordability, and repeatability.

Datasets

  • ImageNet: A large-scale visual database used for the Image Classification benchmark.
  • Extreme Weather Analytics (EWA) Dataset: A dataset comprising 26 years of climate data, used for the EWA benchmark.

Key Insights and Ideas

  • Communication is King (for some workloads): For workloads like EWA, communication bandwidth is a significant bottleneck, and optimizing it can yield substantial performance gains.
  • Computation is King (for others): For workloads like Image Classification, computation efficiency is paramount, and optimizations like mixed-precision training offer significant speedups.
  • Standardization is Crucial: The lack of standardized rules for hyperparameter settings, communication libraries, and other system components leads to unfair comparisons and misleading results.
  • Quality Matters: Simply achieving high FLOPS is insufficient; the benchmark metrics must account for the achieved model quality, penalizing systems that sacrifice accuracy for speed.
  • Roofline Models for AI: Adapting the traditional Roofline model to incorporate communication intensity (COI) provides a powerful tool for understanding HPC AI system performance and identifying optimization opportunities.

Key Results

  • The HPC AI500 V2.0 framework provides a scalable, simple, and affordable way to measure, optimize, and rank HPC AI systems.
  • The proposed benchmarks and metrics were evaluated on an HPC AI system, demonstrating their ability to identify performance bottlenecks and guide optimizations.
  • For Image Classification, mixed-precision optimization and communication compression achieved a peak performance of 939 TFLOPS on 64 GPUs.
  • For EWA, communication compression achieved a speedup of up to 1.6x, with the highest performance reaching 109 TFLOPS.
  • The VFLOPS metric was used to rank systems, showing a highest performance of 642 TVFLOPS for Image Classification on 64 GPUs using mixed precision.

Limitations and Future Work

  • Future Work: The authors plan to explore mixed-precision optimization for EWA in the future. They also intend to update the benchmark epochs and target qualities as needed.
  • Implicit Limitation: While the paper emphasizes the importance of equivalence, achieving perfect equivalence across vastly different hardware and software configurations remains an ongoing challenge.
  • Implicit Limitation: The affordability aspect is relative; running these benchmarks at scale can still be resource-intensive.

Summary updated

BibTeX
@misc{jiang2020hpc,
  author = {Zihan Jiang and Lei Wang and Xingwang Xiong and Wanling Gao and Chunjie Luo and Fei Tang and Chuanxin Lan and Hongxiao Li and Jianfeng Zhan},
  title = {{HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems}},
  year = {2020},
  eprint = {2007.00279},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.00279},
}
Figure