Preliminary report: Initial evaluation of StdPar implementations on AMD GPUs for HPC

Wei-Chen Lin, Simon McIntosh-Smith, Tom DeakinarXiv 2024 · DOI · arXiv

Introduction

The quest for vendor-neutral high-performance computing (HPC) has led to the emergence of the C++17 Parallel Standard Library (StdPar). By allowing standard C++ algorithms to be automatically offloaded to accelerators, StdPar promises a "write once, run anywhere" paradigm. However, achieving performance parity with vendor-specific models like HIP on AMD hardware remains challenging, primarily due to inconsistent support for Unified Shared Memory (USM). This report evaluates current StdPar implementations on AMD GPUs and introduces UTPX, a novel userspace shim designed to overcome hardware-level memory management bottlenecks.

The USM Bottleneck and UTPX

A fundamental requirement for StdPar is a unified address space between the CPU and GPU. On AMD hardware, this functionality is tied to XNACK, a feature that enables hardware-assisted page faulting. In many production environments, XNACK is either disabled, buggy, or poorly documented. When XNACK is unavailable, memory performance often reverts to slow PCIe transfer speeds, rendering StdPar ineffective for bandwidth-intensive applications.

To address this, we developed the Userspace Transparent Paging Extension (UTPX). UTPX is an LD_PRELOAD shim that intercepts memory allocation calls. Instead of relying on kernel-level page faulting, UTPX manages "mirror" allocations and utilizes mprotect alongside custom signal handlers to trigger page migrations in userspace.

The UTPX userspace transparent paging mechanism, showing how signal handlers and mprotect manage memory staging.

Experimental Methodology

We evaluated three primary pathways for StdPar on AMD GPUs: AdaptiveCpp (formerly hipSYCL), ROCm StdPar (rocStdPar), and Intel DPC++ (via vendor plugins). Our benchmarking suite included several HPC "mini-apps":

  • BabelStream: For measuring memory bandwidth efficiency.
  • miniBUDE: A compute-bound molecular docking benchmark.
  • CloverLeaf and TeaLeaf: Complex hydrodynamics and heat conduction codes.

Tests were conducted across diverse AMD architectures: the HPC-grade Instinct MI100, the consumer-grade Radeon VII, and the Ryzen 7950X "Raphael" APU.

Results and Discussion

Our findings indicate that StdPar is a viable alternative to explicit programming models. When memory management is optimized, StdPar implementations achieve nearly 80% of theoretical peak bandwidth, matching the performance of HIP and OpenMP.

A key highlight is the effectiveness of UTPX. On systems where XNACK is disabled (XNACK=0), standard USM models suffer from severe performance degradation. As shown in the performance heatmaps for the MI100, UTPX (specifically the MIRROR and ADVICE modes) successfully restores performance to levels comparable with hardware-assisted XNACK=1 or explicit memory management.

Performance heatmaps for BabelStream on MI100. Note the performance recovery in the UTPX rows compared to the degraded USM XNACK=0 rows.

Furthermore, our evaluation of the Raphael APU revealed that integrated architectures—where CPU and GPU physically share memory—may be the ideal environment for StdPar. On this platform, StdPar reached 90% of HIP's performance without the complexities of discrete memory migration.

Conclusion and Future Work

While the AMD ROCm ecosystem currently presents some friction—including API documentation gaps and kernel stability issues—StdPar demonstrates significant promise for portable HPC. UTPX serves as a critical bridge, providing a high-performance software fallback for systems with limited hardware USM support. Future work will focus on expanding UTPX to support complex data structures with deep pointer indirection and validating these results on newer hardware like the MI250X.

Summary updated

Abstract

Recently, AMD platforms have not supported offloading C++17 PSTL (StdPar) programs to the GPU. Our previous work highlights how StdPar is able to achieve good performance across NVIDIA and Intel GPU platforms. In that work, we acknowledged AMD's past effort such as HCC, which unfortunately is deprecated and does not support newer hardware platforms. Recent developments by AMD, Codeplay, and AdaptiveCpp (previously known as hipSYCL or OpenSYCL) have enabled multiple paths for StdPar programs to run on AMD GPUs. This informal report discusses our experiences and evaluation of currently available StdPar implementations for AMD GPUs. We conduct benchmarks using our suite of HPC mini-apps with ports in many heterogeneous programming models, including StdPar. We then compare the performance of StdPar, using all available StdPar compilers, to contemporary heterogeneous programming models supported on AMD GPUs: HIP, OpenCL, Thrust, Kokkos, OpenMP, SYCL. Where appropriate, we discuss issues encountered and workarounds applied during our evaluation. Finally, the StdPar model discussed in this report largely depends on Unified Shared Memory (USM) performance and very few AMD GPUs have proper support for this feature. As such, this report demonstrates a proof-of-concept host-side userspace pagefault solution for models that use the HIP API. We discuss performance improvements achieved with our solution using the same set of benchmarks.
BibTeX
@misc{lin2024preliminary,
  author = {Wei-Chen Lin and Simon McIntosh-Smith and Tom Deakin},
  title = {{Preliminary report: Initial evaluation of StdPar implementations on AMD GPUs for HPC}},
  year = {2024},
  eprint = {2401.02680},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.02680},
}
Figure