Understanding Data Movement in Tightly Coupled Heterogeneous Systems: A Case Study with the Grace Hopper Superchip

Luigi Fusco, Mikhail Khalilov, Marcin Chrapek, Giridhar Chukkapalli, Thomas C. Schulthess, Torsten HoeflerarXiv 2024 · DOI · arXiv

Abstract

The NVIDIA GH200 Grace Hopper Superchip integrates an ARM-based Grace CPU and a Hopper GPU via the high-speed NVLink-C2C interconnect to address the "memory wall" in HPC and AI. This paper provides a comprehensive performance characterization of the GH200 on the "Alps" supercomputer, analyzing bandwidth, latency, and real-world application performance. Results demonstrate that while the 900 GB/s C2C link significantly improves CPU-GPU communication over traditional PCIe, strategic data placement remains a critical factor for performance, as evidenced by a 10x penalty in LLM inference when using LPDDR5 instead of HBM3.

Introduction

In High-Performance Computing (HPC) and AI, compute power has historically outpaced data movement speeds, leading to a "memory wall." Traditional systems rely on the PCIe bus, which often acts as a bottleneck for data-hungry GPUs. The NVIDIA GH200 "Superchip" addresses this by fusing an ARM-based Grace CPU and a Hopper GPU on a single module using the high-speed NVLink-C2C interconnect. This study investigates the performance nuances of this tightly coupled architecture, focusing on unified memory behavior and the costs of data movement across different memory tiers (LPDDR5 vs. HBM3).

Architecture and Methodology

The research was conducted on the Alps supercomputer at the Swiss National Supercomputing Centre (CSCS), utilizing a "Quad GH200" node configuration. This setup connects four Superchips through a combination of NVLink and Grace Interconnects, creating a complex NUMA (Non-Uniform Memory Access) environment.

Architectural overview of the GH200 Superchip and its interconnects within a Quad-node configuration.

The evaluation methodology employed custom microbenchmarks to measure raw bandwidth and latency for read, write, and copy operations across all permutations of CPU/GPU and memory locations. Latency was probed using "Pointer Chase" and "Ping-Pong" benchmarks, while application-level testing included General Matrix Multiplication (GEMM), Large Language Model (LLM) inference (Llama 2), and collective communications via NCCL.

Performance Characterization

The core of the GH200's performance lies in the NVLink-C2C interconnect, providing a bidirectional bandwidth of 900 GB/s—roughly 7x faster than PCIe Gen5. This high-speed link effectively extends the GPU's memory space into the CPU's LPDDR5.

Latency vs. Bandwidth characteristics for Grace and Hopper processors across local and remote memory types.

Key findings from the characterization include:

  • Memory Asymmetry: Data movement throughput varies by direction; for instance, the Grace CPU achieves higher throughput copying to a peer Superchip than receiving from one.
  • Cache Behavior: A surprising discovery was that the Hopper GPU's L2 cache can cache data residing in the HBM memory of a different Superchip. This makes remote HBM access significantly faster than local DDR access in specific cached scenarios.
  • The "First Touch" Policy: In the unified address space, physical data location is often determined by the first processor to access it. Improper management can lead to data residing in slower LPDDR5 instead of the lightning-fast HBM3.

Key Results

  • Bandwidth and Latency: Local HBM3 bandwidth reaches a staggering 4 TB/s with a latency of ~344ns. Crossing the C2C link to access remote memory increases latency to approximately 800-1000ns.
  • LLM Inference: Performance is highly sensitive to memory placement. For Llama2-7b, using HBM results in an inference time of 3.6 seconds for 100 tokens. Moving the model to the Grace CPU's LPDDR5 memory increases this to 33.7 seconds, a nearly 10x performance penalty.
  • GEMM Sensitivity: Matrix multiplication performance depends heavily on the storage location of source matrices (A and B), while the destination matrix (C) placement has a negligible impact.

Conclusion

The NVIDIA GH200 represents a significant step toward overcoming the memory wall through tight CPU-GPU integration. However, the unified memory space does not eliminate the need for careful data orchestration. Future work will focus on scaling these findings to larger clusters, exploring automatic data migration features in newer CUDA versions, and evaluating the power efficiency of the C2C interconnect compared to traditional architectures.

Summary updated

Abstract

Heterogeneous supercomputers have become the standard in HPC. GPUs in particular have dominated the accelerator landscape, offering unprecedented performance in parallel workloads and unlocking new possibilities in fields like AI and climate modeling. With many workloads becoming memory-bound, improving the communication latency and bandwidth within the system has become a main driver in the development of new architectures. The Grace Hopper Superchip (GH200) is a significant step in the direction of tightly coupled heterogeneous systems, in which all CPUs and GPUs share a unified address space and support transparent fine grained access to all main memory on the system. We characterize both intra- and inter-node memory operations on the Quad GH200 nodes of the new Swiss National Supercomputing Centre Alps supercomputer, and show the importance of careful memory placement on example workloads, highlighting tradeoffs and opportunities.
BibTeX
@misc{fusco2024understanding,
  author = {Luigi Fusco and Mikhail Khalilov and Marcin Chrapek and Giridhar Chukkapalli and Thomas C. Schulthess and Torsten Hoefler},
  title = {{Understanding Data Movement in Tightly Coupled Heterogeneous Systems: A Case Study with the Grace Hopper Superchip}},
  year = {2024},
  eprint = {2408.11556},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.11556},
}
Figure