Abstract
The NVIDIA GH200 Grace Hopper Superchip integrates an ARM-based Grace CPU and a Hopper GPU via the high-speed NVLink-C2C interconnect to address the "memory wall" in HPC and AI. This paper provides a comprehensive performance characterization of the GH200 on the "Alps" supercomputer, analyzing bandwidth, latency, and real-world application performance. Results demonstrate that while the 900 GB/s C2C link significantly improves CPU-GPU communication over traditional PCIe, strategic data placement remains a critical factor for performance, as evidenced by a 10x penalty in LLM inference when using LPDDR5 instead of HBM3.
Introduction
In High-Performance Computing (HPC) and AI, compute power has historically outpaced data movement speeds, leading to a "memory wall." Traditional systems rely on the PCIe bus, which often acts as a bottleneck for data-hungry GPUs. The NVIDIA GH200 "Superchip" addresses this by fusing an ARM-based Grace CPU and a Hopper GPU on a single module using the high-speed NVLink-C2C interconnect. This study investigates the performance nuances of this tightly coupled architecture, focusing on unified memory behavior and the costs of data movement across different memory tiers (LPDDR5 vs. HBM3).
Architecture and Methodology
The research was conducted on the Alps supercomputer at the Swiss National Supercomputing Centre (CSCS), utilizing a "Quad GH200" node configuration. This setup connects four Superchips through a combination of NVLink and Grace Interconnects, creating a complex NUMA (Non-Uniform Memory Access) environment.
The evaluation methodology employed custom microbenchmarks to measure raw bandwidth and latency for read, write, and copy operations across all permutations of CPU/GPU and memory locations. Latency was probed using "Pointer Chase" and "Ping-Pong" benchmarks, while application-level testing included General Matrix Multiplication (GEMM), Large Language Model (LLM) inference (Llama 2), and collective communications via NCCL.
Performance Characterization
The core of the GH200's performance lies in the NVLink-C2C interconnect, providing a bidirectional bandwidth of 900 GB/s—roughly 7x faster than PCIe Gen5. This high-speed link effectively extends the GPU's memory space into the CPU's LPDDR5.
Key findings from the characterization include:
- Memory Asymmetry: Data movement throughput varies by direction; for instance, the Grace CPU achieves higher throughput copying to a peer Superchip than receiving from one.
- Cache Behavior: A surprising discovery was that the Hopper GPU's L2 cache can cache data residing in the HBM memory of a different Superchip. This makes remote HBM access significantly faster than local DDR access in specific cached scenarios.
- The "First Touch" Policy: In the unified address space, physical data location is often determined by the first processor to access it. Improper management can lead to data residing in slower LPDDR5 instead of the lightning-fast HBM3.
Key Results
- Bandwidth and Latency: Local HBM3 bandwidth reaches a staggering 4 TB/s with a latency of ~344ns. Crossing the C2C link to access remote memory increases latency to approximately 800-1000ns.
- LLM Inference: Performance is highly sensitive to memory placement. For Llama2-7b, using HBM results in an inference time of 3.6 seconds for 100 tokens. Moving the model to the Grace CPU's LPDDR5 memory increases this to 33.7 seconds, a nearly 10x performance penalty.
- GEMM Sensitivity: Matrix multiplication performance depends heavily on the storage location of source matrices (A and B), while the destination matrix (C) placement has a negligible impact.
Conclusion
The NVIDIA GH200 represents a significant step toward overcoming the memory wall through tight CPU-GPU integration. However, the unified memory space does not eliminate the need for careful data orchestration. Future work will focus on scaling these findings to larger clusters, exploring automatic data migration features in newer CUDA versions, and evaluating the power efficiency of the C2C interconnect compared to traditional architectures.
Summary updated