Benchmarking and Dissecting the Nvidia Hopper GPU Architecture

Weile Luo, Ruibo Fan, Zeyu Li, Dayou Du, Qiang Wang, Xiaowen Chu

Introduction

Nvidia’s Hopper architecture (H100/H800) represents a significant leap in GPU design, yet many of its low-level operational details remain proprietary. This study provides an independent microarchitectural analysis, "peeling back the sticker" to evaluate how the hardware handles instructions, memory hierarchy, and the new FP8 precision format. The research reveals that achieving peak performance requires a fundamental shift in programming models, specifically moving toward asynchronous warp-group operations.

Architectural Overview

The Hopper architecture introduces several key hardware components designed to accelerate AI and high-performance computing (HPC) workloads. Central to this is the Streaming Multiprocessor (SM), which now features 4th Generation Tensor Cores, a Tensor Memory Accelerator (TMA), and significantly expanded L1/Shared Memory capacity.

Nvidia Hopper Architecture Overview

Methodology

The researchers utilized a multi-faceted benchmarking approach to isolate Hopper-specific improvements:

  • Microbenchmarking: Pointer-chasing (P-Chase) was used to measure latency and throughput across L1, L2, and Global memory.
  • Instruction Analysis: SASS disassembly allowed the team to observe the execution of machine-level assembly, particularly the new wgmma (Warp Group Matrix Multiply-Accumulate) instructions.
  • Comparative Evaluation: Identical tests were conducted across Ampere (A100), Ada Lovelace (RTX 4090), and Hopper (H800) architectures.
  • Real-world Workloads: Evaluation included Llama model configurations (3B to 70B) and ShareGPT datasets to simulate realistic Large Language Model (LLM) inference scenarios.

The wgmma Revolution and Instruction Evolution

A primary finding of the study is the transition from traditional mma (warp-level) to wgmma (warp-group level) instructions. While legacy mma instructions are synchronous, wgmma is asynchronous, allowing the GPU to "fire and forget" matrix math operations while threads perform other tasks.

Comparison of mma and wgmma Instruction Formats

The study highlights an "MMA Trap": using legacy mma instructions on Hopper yields only ~63% of theoretical peak performance. To exceed 95% utilization, developers must adopt the wgmma paradigm. Furthermore, Hopper introduces "Dynamic Programming X" (DPX) instructions, providing hardware acceleration for min/max operations common in bioinformatics, resulting in up to 13x speedups over the A100 for specific algorithms like Smith-Waterman.

Memory and Performance Results

Hopper demonstrates substantial gains in memory subsystem efficiency. The H800’s L2 cache throughput is 2.2x faster than its predecessors. Additionally, the new Distributed Shared Memory (DSM) enables direct SM-to-SM communication, reducing data exchange latency by 32% compared to L2-mediated transfers.

Regarding FP8 performance, while the theoretical throughput is massive, practical gains are sensitive to matrix size. For smaller matrices, the overhead of quantization and scaling (converting FP16 to FP8) can negate the speedup. Significant benefits only emerge when the model's hidden size exceeds 8192. In memory-bound LLM inference tasks, the computational advantage of FP8 is often secondary to HBM bandwidth limitations.

Conclusion and Future Outlook

The Hopper architecture is a "green" powerhouse, offering 1.6x to 1.69x better performance-per-watt than the Ampere generation. However, software maturity remains a hurdle; Nvidia’s Transformer Engine is still evolving, and "conversion islands" between FP16 and FP8 can slow down execution pipelines. This dissection serves as a critical guide for developers aiming to extract maximum performance from the H100 series by leveraging its new asynchronous capabilities and specialized hardware units.

Summary updated

BibTeX
@inproceedings{luo2024benchmarking,
  author = {Weile Luo and Ruibo Fan and Zeyu Li and Dayou Du and Qiang Wang and Xiaowen Chu},
  title = {{Benchmarking and Dissecting the Nvidia Hopper GPU Architecture}},
  booktitle = {IPDPS},
  year = {2024},
  doi = {10.1109/ipdps57955.2024.00064},
}
Figure