Experiences with OpenMP in tmLQCD

Albert Deuzeman, Karl Jansen, Bartosz Kostrzewa, Carsten UrbacharXiv 2013 · OpenReview

TL;DR

This paper details the practical challenges and lessons learned from parallelizing a lattice Quantum Chromodynamics (QCD) code, tmLQCD, using OpenMP for multi-threading. The authors provide valuable insights into optimizing performance, managing memory, and handling race conditions in complex scientific simulations, offering a roadmap for others undertaking similar parallelization efforts.

What Problem Does This Paper Address?

Lattice Quantum Chromodynamics (QCD) simulations are computationally intensive, requiring massive parallelization to tackle complex calculations. While Message Passing Interface (MPI) has been the standard for distributed-memory parallelism, the increasing core counts in modern CPUs necessitate exploring shared-memory parallelism within nodes. This paper addresses the practical difficulties of integrating OpenMP, a shared-memory parallel programming model, into an existing, large-scale lattice QCD code (tmLQCD) to achieve better performance and scalability. Specifically, it investigates how to effectively manage data, optimize thread distribution, and overcome synchronization issues that arise when combining OpenMP with MPI in a hybrid approach.

Methods

The authors introduce OpenMP directives into the tmLQCD software suite, focusing on the computationally demanding "hopping matrix" kernel. They employ several key strategies:

  • Variable Scoping: Leveraging C++'s variable scoping rules to automatically manage thread-local and thread-global data, simplifying code evolution and reducing the risk of errors from manual list management.
  • Performance Profiling: Using tools like Intel® VTune™ Performance Analyzer and SCALASCA to identify performance bottlenecks, such as cache misses and thread idling.
  • Optimization Techniques: Implementing optimizations like precomputing lattice indices to improve cache access patterns and exploring different thread distribution strategies (e.g., threads per process) to match hardware capabilities.
  • Overlapping Communication and Computation: Investigating techniques to hide communication latency by performing computation concurrently, using nowait directives and different scheduling strategies (static vs. guided).
  • Race Condition Management: Discussing strategies for handling race conditions, including the use of atomic directives and the concept of "halo memory" to ensure data integrity when multiple threads access shared data.

Datasets

The paper does not explicitly list or introduce new datasets. Instead, it uses the tmLQCD benchmark application, which involves simulating lattice QCD configurations on various lattice volumes (e.g., 12⁴, 8³, 48³ x 96). The performance measurements are conducted on specific hardware platforms: Intel® Xeon X5560 Infiniband cluster, Cray XC30, and BlueGene/Q (JuQueen) at JSC Juelich, and SuperMUC.

Key Insights and Ideas

  • Scoping Rules Simplify, But Can Obscure: While using C++ scoping rules for thread-local/global data is convenient and reduces errors, it can make it harder to reason about complex parallelism and introduce "orphaned" directives.
  • Cache Performance is Crucial: A significant performance boost (up to 20%) was achieved by optimizing memory access patterns to improve cache utilization, highlighting the importance of data locality in scientific codes.
  • Thread Distribution Matters: The optimal number of threads per process and processes per node depends heavily on the local lattice volume and whether it fits within the CPU cache. For cache-resident data, more threads per process are beneficial; for smaller volumes, minimizing overhead by using more processes with fewer threads is better.
  • Overlapping Communication is Tricky: Simply attempting to overlap communication and computation with standard MPI calls doesn't always work as expected. Careful implementation, often involving nowait directives and specific scheduling, is required to achieve actual latency hiding.
  • Race Condition Solutions Have Trade-offs: While atomic directives are generally preferred over critical sections due to lower overhead, their correctness can be challenging to guarantee in complex scenarios. "Halo memory" offers a safer alternative but requires more memory and an extra loop for accumulation.
  • Overhead is Machine-Dependent: OpenMP overheads can vary significantly between architectures, with some machines showing substantial thread forking overhead even when thread recycling is expected.

Key Results

  • Hybrid Performance: On the BlueGene/Q, the hybrid OpenMP/MPI approach consistently outperforms pure MPI for the hopping matrix kernel, especially when communication and computation are effectively overlapped.
  • Cache Optimization Impact: A 20% performance improvement was observed for the hopping matrix kernel after optimizing memory access for cache efficiency.
  • OpenMP Overhead: On BG/Q, OpenMP overhead impacts pure floating-point performance by approximately 11 GFlop/s per node.
  • Scaling Behavior: For cache-resident local lattice volumes, 1 process per node with 64 threads was found to be fastest. For smaller volumes, configurations with 4 threads per process or 32 threads total were more beneficial.
  • Overlapping Benefits: The hybrid code with overlapping communication and computation showed noticeable performance improvements, with specific scheduling strategies (like guided) yielding better results than static scheduling.

Limitations and Future Work

  • MPI Parameter Tuning: The authors suggest that further optimization of MPI parameters might improve the performance of the hybrid MPI/OpenMP version on certain architectures.
  • RDMA Implementation: An RDMA implementation using one-sided MPI communicators was not attempted but is suggested as a future direction.
  • Fine-grained Overhead Measurement: More detailed measurements of overheads using SCALASCA event tracing are planned.
  • Atomic Directive Study: A dedicated study on the potential for errors with atomic statements in specific contexts is planned.
  • Halo Memory for Derivatives: Testing halo memory for the fermionic derivative calculation is also a future consideration.
  • Architecture Dependence: The performance benefits and challenges of OpenMP integration are shown to be highly dependent on the underlying hardware architecture, suggesting that optimizations may need to be tailored for each platform.

Summary updated

Abstract

An overview is given of the lessons learned from the introduction of multi-threading using OpenMP in tmLQCD. In particular, programming style, performance measurements, cache misses, scaling, thread distribution for hybrid codes, race conditions, the overlapping of communication and computation and the measurement and reduction of certain overheads are discussed. Performance measurements and sampling profiles are given for different implementations of the hopping matrix computational kernel.
BibTeX
@misc{deuzeman2013experiences,
  author = {Albert Deuzeman and Karl Jansen and Bartosz Kostrzewa and Carsten Urbach},
  title = {{Experiences with OpenMP in tmLQCD}},
  year = {2013},
  eprint = {1311.4521},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1311.4521},
}
Figure