Abstract
With multi-core processors a ubiquitous building block of modern supercomputers, it is now past time to enable applications to embrace these developments in processor design. To achieve exascale performance, applications will need ways of exploiting the new levels of parallelism that are exposed in modern high-performance computers. A typical approach to this is to use shared-memory programming techniques to best exploit multi-core nodes along with inter-node message passing. In this paper, we describe the addition of OpenMP threaded functionality to the PETSc library. We highlight some issues that hinder good performance of threaded applications on modern processors and describe how to negate them. The OpenMP branch of PETSc was benchmarked using matrices extracted from Fluidity, a CFD application code, which uses the library as its linear solver engine. The overall performance of the mixed-mode implementation is shown to be superior to that of the pure-MPI version.
Introduction: The Challenge of Fat Nodes
High-performance computing (HPC) has traditionally relied on the Message Passing Interface (MPI) to distribute workloads. However, the shift toward "fat nodes"—single machines containing dozens of CPU cores sharing a memory pool—presents new challenges. In a pure MPI model, every core acts as an independent process, leading to significant communication overhead and synchronization bottlenecks at exascale.
Furthermore, modern hardware often utilizes Non-Uniform Memory Access (NUMA). In these architectures, memory access latency depends on the physical distance between a processor core and the memory bank. Pure MPI often fails to account for this hierarchy, leading to inefficient memory bandwidth utilization.
Mixed-Mode Implementation
To address these issues, the authors integrated shared-memory threading via OpenMP into the Portable Extensible Toolkit for Scientific Computation (PETSc). This "mixed-mode" approach uses MPI for inter-node communication and OpenMP for intra-node parallelism, allowing the library to better exploit the shared memory of modern nodes.
The implementation focused on the core Vector (Vec) and Matrix (Mat) classes. Key technical strategies included:
- Macro-based Threading: Custom macros toggle threading based on data size, avoiding the "threading tax" (overhead) for small calculations.
- NUMA-Awareness & First-Touch: Utilizing the Linux "first-touch" policy ensures data is physically located in the memory bank closest to the thread processing it, maximizing bandwidth.
- Affinity Management: Explicitly pinning threads to specific cores prevents the operating system from migrating tasks, which would otherwise destroy cache performance.
Performance Evaluation
The implementation was benchmarked using matrices from Fluidity, a Computational Fluid Dynamics (CFD) application. Test cases ranged from standard 3D turbulence problems to the "Flue" matrix, which contains nearly 750 million non-zero elements.
The results demonstrate that the mixed-mode implementation consistently outperforms pure MPI at scale. On a 32-core AMD Interlagos node, memory bandwidth doubled (from ~21 GB/s to ~43 GB/s) due to proper affinity management. Most significantly, for the massive "Flue" matrix, the hybrid version achieved a 50% performance improvement over pure MPI when running on 8,192 cores.
Conclusion
The integration of OpenMP into PETSc provides a scalable path forward for linear algebra on modern multi-core architectures. By reducing communication overhead and optimizing for NUMA, the hybrid model achieves superior performance and energy efficiency compared to traditional pure-MPI approaches. Future work will focus on threading more complex preconditioners and further refining data locality.
Summary updated