Blocking Versus Non-Blocking Halo Exchange

Anthony BourachedarXiv 2017 · arXiv

Introduction

The Lattice Boltzmann Method (LBM) has emerged as a robust framework for simulating complex fluid dynamics. However, its implementation on high-performance computing (HPC) architectures is frequently constrained by memory bandwidth and communication overhead. To handle large-scale systems, researchers employ domain decomposition, partitioning a global 3D grid into smaller subdomains assigned to individual MPI tasks.

The Halo Exchange Challenge

A critical component of parallel LBM is the "halo exchange." In every simulation time step, each subdomain must synchronize its boundary data—known as halos—with its adjacent neighbors. In a 3D decomposition, a subdomain typically interacts with 26 neighbors, including 6 faces, 12 edges, and 8 corners.

Visualization of domain decomposition in a 3D grid, where a central subdomain (blue) must exchange halo data with its neighbors (red).

The standard implementation in many production codes, such as the "Ludwig" framework, utilizes a blocking communication strategy. This approach typically proceeds in three sequential stages (X-direction, then Y, then Z). By carrying edge and corner data along these primary axes, the number of messages is reduced to six. However, this serial dependency forces processors to remain idle while waiting for network transfers to complete, creating a significant synchronization bottleneck.

Non-Blocking Communication and Work-Communication Overlap

This study investigates a non-blocking alternative designed to mitigate processor idle time. By utilizing MPI_Issend and MPI_Irecv, the routine initiates all 26 neighbor communications simultaneously. This strategy shifts the performance focus from minimizing message count to maximizing network throughput and enabling "work-communication overlap."

To demonstrate the potential of this overlap, the researchers reordered the simulation workflow. A specific task—build_update_map, which identifies solid particles within the fluid—was moved into the temporal window between the initiation and completion of the non-blocking communication. This allows the CPU to perform meaningful computational work while the network hardware handles data transit in the background.

Experimental Methodology

The implementation was benchmarked on the ARCHER supercomputer (a Cray XC30 system) using up to 24,576 MPI tasks. The testing suite included:

  • Grid Configurations: Cubic lattices ranging from $96^3$ to $768^3$ sites.
  • Performance Metrics: Effective bandwidth and strong scaling efficiency.
  • Verification: Custom unit tests ensured bit-for-bit data consistency between the new non-blocking routine and the established blocking baseline.

Key Insights and Results

The research highlights a fundamental trade-off between latency and bandwidth. While sending 26 messages increases the initial latency overhead compared to the 6-message blocking approach, the non-blocking method excels as message sizes grow.

Performance Gains

The non-blocking version demonstrated superior performance for larger subdomains (e.g., $58^3$ to $88^3$ lattice sites). In these scenarios, the ability to saturate MPI bandwidth—which occurs at approximately 0.5 MB on the ARCHER system—outweighs the latency costs of managing additional messages.

Overlap Efficiency

In simulations of a $256 \times 256 \times 128$ system, the non-blocking strategy with work-communication overlap consistently outperformed both the standard blocking version and the "raw" non-blocking version without overlap. This confirms that hiding communication costs behind independent computation is a viable path to increased efficiency.

Scaling and Geometry

The non-blocking approach proved particularly resilient for "dimensionally imbalanced" (non-cubic) subdomains. It maintained competitive parallel efficiency in strong scaling tests up to 6,144 cores, suggesting it is a robust alternative for massive-scale scientific computing.

Conclusion and Future Directions

The transition from blocking to non-blocking halo exchange offers a clear performance advantage in LBM simulations, provided the subdomains are sufficiently large or there is independent work to overlap. However, the benefits are problem-dependent; simulations lacking independent computational tasks during the exchange window may see diminishing returns. Future optimizations could involve "asynchronous unpacking," where halo buffers are processed as soon as individual messages arrive rather than waiting for the entire set of 26 transfers to conclude.

Summary updated

Abstract

This report describes the design, implementation and analysis of a non-blocking halo exchange routine as an alternative to the blocking halo exchange routine in the lattice Boltzmann code Ludwig. The alternative, non-blocking, routine is implemented in such a way to allow work-communication overlap. Detailed benchmarks in this report show that the non-blocking version is a good alternative even without any work-communication overlap. Work-Communication overlap can be used to improve the performance of the non-blocking routine. Development and benchmarking were conducted on the UK national supercomputer, ARCHER.
BibTeX
@misc{bourached2017blocking,
  author = {Anthony Bourached},
  title = {{Blocking Versus Non-Blocking Halo Exchange}},
  year = {2017},
  eprint = {1709.06175},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.06175},
}
Figure