Communication Optimization for Multiphase Flow Solver in the Library of OpenFOAM

Zhipeng Lin, Wenjing Yang, Houcun Zhou, Xinhai Xu, Liaoyuan Sun, Yongjun Zhang, Yuhua TangWater 2018 · DOI

TL;DR

This paper tackles the scalability bottleneck in OpenFOAM's multiphase flow solvers, which is primarily caused by inefficient communication between computational cores. The researchers developed optimized versions of key solver modules (MULES and PCG) that significantly reduce communication overhead, leading to dramatic improvements in parallel scalability and solving speed without sacrificing accuracy.

What Problem Does This Paper Address?

OpenFOAM is a powerful open-source library for Computational Fluid Dynamics (CFD), widely used in various engineering fields. Multiphase flow simulations, crucial for many applications, are a significant part of OpenFOAM's capabilities. However, a major challenge with these solvers, especially when running on large parallel computing clusters, is their limited scalability and slow performance compared to commercial alternatives. The authors identify that this performance degradation stems from "communication overhead" – the time spent by different processing cores exchanging data. Specifically, they pinpoint two core modules, the Multidimensional Universal Limiter for Explicit Solution (MULES) and the Preconditioned Conjugate Gradient (PCG) algorithm, as the primary culprits for this communication bottleneck.

Methods

The researchers employed a systematic approach to identify and address the communication issues:

  • Bottleneck Identification: They first conducted scalability tests on a standard damBreak multiphase flow case using profiling tools to pinpoint the exact modules causing performance degradation. This revealed that MPI (Message Passing Interface) communication within MULES and PCG was the main bottleneck.
  • Communication Analysis: A deeper dive into the communication patterns of MULES and PCG uncovered specific inefficiencies:
  • MULES: Redundant communication was identified, particularly in the combineReduce() function, which was deemed unnecessary for static mesh simulations.
  • PCG: Global synchronization, a common issue in iterative solvers, was found to be a major performance limiter.
  • Optimization Algorithms: Based on this analysis, they developed several optimized algorithms:
  • optMULES: This optimization focuses on removing the redundant combineReduce() calls in MULES by intelligently memorizing and reusing message sizes.
  • OFPiPePCG: This solver rearranges the PCG algorithm, introducing intermediate variables and overlapping computations with non-blocking communication to reduce global synchronization.
  • OFRePiPePCG: This further refines OFPiPePCG by overlapping matrix-vector multiply and vector update operations with non-blocking computations, aiming to hide communication latency even more effectively.
  • Implementation: These optimizations were integrated into the OpenFOAM library, ensuring compatibility and ease of use.

Datasets

The paper does not introduce new datasets. Instead, it utilizes established benchmark cases commonly used in CFD research and within the OpenFOAM community:

  • cavity case: A 2D lid-driven cavity flow simulation.
  • damBreak case: A 2D simulation of water rushing to a dam.
  • head-form case: A 3D simulation of cavitation around a cylinder.
  • Real-life 3D dam break flood: A more complex, real-world application case.

These cases were chosen to represent a diverse range of simulation complexities and to test the optimizations under various conditions.

Key Insights and Ideas

  • MULES' Hidden Redundancy: The discovery that combineReduce() in MULES is often redundant for static mesh simulations is a significant finding. This highlights how seemingly small inefficiencies can accumulate and severely impact performance at scale.
  • PCG's Global Synchronization Challenge: The paper reinforces the well-known challenge of global synchronization in iterative solvers like PCG, but it provides a concrete analysis of how this impacts OpenFOAM's multiphase solvers.
  • Overlapping Communication and Computation: The core idea of overlapping communication with computation, particularly using non-blocking MPI calls, is crucial. This allows processors to perform useful work while waiting for data from other cores, effectively hiding communication latency.
  • Targeted Optimization: The success of optimizing specific modules (MULES and PCG) rather than a complete overhaul demonstrates the power of targeted performance tuning.

Key Results

The experimental results demonstrate substantial improvements:

  • Scalability: The optimized solvers dramatically increased parallel scalability. For the damBreak case, strong scalability was improved by approximately three times.
  • Speedup: Execution times were significantly reduced, with reductions of up to 63.87% observed in test cases.
  • Accuracy: The optimizations were achieved with minimal to no loss of accuracy, with relative errors being very small (less than 1% for PCG optimizations).
  • 3D Performance: The optimizations showed effectiveness even in complex 3D simulations like the head-form case.
  • MULES vs. PCG Optimization: The study suggests that eliminating redundant communication in MULES (optMULES) has a more significant impact on weak scalability than rearranging the PCG algorithm alone.

Limitations and Future Work

  • Hardware Dependency: The paper acknowledges that the effectiveness of non-blocking communication can be hardware-dependent. In environments not optimized for non-blocking operations, the performance gains might be less pronounced.
  • Generalizability of MULES Optimization: While the MULES optimization is presented as specific to OpenFOAM, the authors suggest its underlying idea might be applicable to other VOF-based CFD software.
  • Future Work: The authors implicitly suggest further exploration into optimizing other modules or exploring more advanced non-blocking communication strategies. The paper also mentions that the optimized PCG solvers could potentially work on other C++ codes with modifications, hinting at broader applicability.

Summary updated

Abstract

Multiphase flow solvers are widely-used applications in OpenFOAM, whose scalability suffers from the costly communication overhead. Therefore, we establish communication-optimized multiphase flow solvers in OpenFOAM. In this paper, we first deliver a scalability bottleneck test on the typical multiphase flow case damBreak and reveal that the Message Passing Interface (MPI) communication in a Multidimensional Universal Limiter for Explicit Solution (MULES) and a Preconditioned Conjugate Gradient (PCG) algorithm is the short slab of multiphase flow solvers. Furthermore, an analysis of the communication behavior is carried out. We find that the redundant communication in MULES and the global synchronization in PCG are the performance limiting factors. Based on the analysis, we propose our communication optimization algorithm. For MULES, we remove the redundant communication and obtain optMULES. For PCG, we import several intermediate variables and rearrange PCG to reduce the global communication. We also overlap the computation of matrix-vector multiply and vector update with the non-blocking computation. The resulting algorithms are respectively referred to as OFPiPePCG and OFRePiPePCG. Extensive experiments show that our proposed method could dramatically increase the parallel scalability and solving speed of multiphase flow solvers in OpenFOAM approximately without the loss of accuracy.
BibTeX
@inproceedings{lin2018communication,
  author = {Zhipeng Lin and Wenjing Yang and Houcun Zhou and Xinhai Xu and Liaoyuan Sun and Yongjun Zhang and Yuhua Tang},
  title = {{Communication Optimization for Multiphase Flow Solver in the Library of OpenFOAM}},
  booktitle = {Water},
  year = {2018},
  doi = {10.3390/w10101461},
}
Figure