Asynchronous in Situ Processing with Gromacs: Taking Advantage of GPUs

Monica L. Hernandez, Matthieu Dreher, Carlos J. Barrios, Bruno RaffinCARLA 2015 · DOI

TL;DR

This paper tackles the challenge of efficiently analyzing massive datasets generated by scientific simulations, particularly those leveraging GPUs. The authors propose and evaluate strategies for performing "in situ" analysis (processing data as it's generated, in memory) directly on GPUs within the Gromacs molecular dynamics simulation framework, demonstrating improved resource utilization and performance compared to traditional CPU-based in situ methods.

What Problem Does This Paper Address?

Modern scientific simulations, especially in fields like biology and fluid dynamics, are producing an overwhelming amount of data. Storing and analyzing this data post-simulation is becoming a significant bottleneck, with estimates suggesting that less than 1% of data will be saved in the Exascale era due to I/O limitations. The "in situ" paradigm offers a solution by processing data in memory as it's generated, avoiding the need to write massive files to disk. However, integrating in situ analytics with hybrid simulations that utilize accelerators like GPUs presents new challenges. Specifically, efficiently allocating computational resources between the simulation and the analytics, especially when the simulation itself is GPU-accelerated, is complex. This paper aims to explore and evaluate different strategies for performing in situ analytics on GPUs within the context of Gromacs, a popular molecular dynamics simulation package.

Methods

The core of this work involves instrumenting the Gromacs simulation code to enable asynchronous in situ analytics and then evaluating different placement strategies for these analytics.

  • Gromacs Instrumentation: Gromacs was modified to periodically extract atom positions. This extraction is designed to be lightweight and asynchronous, minimizing its impact on the simulation's performance. The native file writing mechanism of Gromacs was disabled to avoid interference.
  • FlowVR Middleware: The FlowVR middleware was used to manage the communication and execution of the in situ analytics modules. FlowVR allows describing applications as a graph of modules connected by communication channels, enabling asynchronous execution.
  • Placement Strategies: Two primary placement strategies were investigated:
  • Helper Core Strategy: This strategy dedicates a separate CPU core (and potentially a GPU) on each node for in situ analytics, isolating it from the main simulation processes.
  • Overlapping Strategy: This strategy runs the in situ analytics on the same computational resources as the simulation. In this work, it meant sharing a GPU between a simulation process and an in situ analytics task.
  • GPU-Accelerated Analytics: A key novelty is the application of these placement strategies to in situ analytics running on the GPU, rather than solely on the CPU.
  • Benchmarking: To evaluate the impact of in situ analytics, several micro-benchmarks were employed:
  • PI (CPU): Stresses the floating-point units of the CPU.
  • Bandwidth (GPU): Stresses data transfer between CPU and GPU.
  • Matrix Multiplication (GPU): Stresses the computational units of the GPU.

Datasets

Not discussed in the paper. The paper focuses on evaluating the performance of the Gromacs simulation and in situ analytics on a specific cluster, rather than using pre-defined datasets. The "dataset" in this context is the data generated by the Gromacs simulation itself.

Key Insights and Ideas

  • Gromacs Underutilizes GPUs Natively: Even when running in hybrid CPU/GPU mode, Gromacs doesn't fully utilize the available GPU resources. This leaves room for other tasks to be performed on the GPU without significantly impacting simulation performance.
  • CPU Bottleneck is Key: In the tested configuration, the CPU was identified as the bottleneck for Gromacs. This means that stressing the CPU with in situ analytics (like the PI benchmark) had a more significant negative impact on simulation performance than stressing the GPU.
  • GPU Analytics Can Improve Resource Utilization: Performing in situ analytics on the GPU can lead to higher overall GPU utilization, effectively using resources that would otherwise be idle.
  • Placement Strategy Matters: Both helper core and overlapping strategies have trade-offs. Overlapping can be more efficient for light workloads but suffers significantly with heavier loads. Helper core has a higher initial cost but offers better isolation and stability for more demanding analytics.
  • GPU Analytics Offer a Promising Alternative: When the CPU is the simulation bottleneck, offloading analytics to the GPU can be more efficient than running them on the CPU, as it doesn't exacerbate the existing bottleneck.

Key Results

  • Instrumentation Cost is Negligible: The overhead of instrumenting Gromacs for in situ data extraction was found to be very low (around 0.5% increase in simulation time) due to a low output frequency.
  • Hybrid Mode Outperforms CPU-Only: Gromacs in hybrid CPU/GPU mode showed performance improvements of 3.2x to 4.38x compared to CPU-only mode.
  • CPU Analytics Impact: When running PI (CPU benchmark) with the overlapping strategy, simulation performance dropped significantly (over 30%) with increasing PI iterations. The helper core strategy showed more stable behavior, becoming more efficient than overlapping for higher PI iteration counts.
  • GPU Analytics Impact: When running multMatrix (GPU benchmark) with the overlapping strategy, performance dropped up to 20% for heavy workloads, but GPU utilization increased up to 99%. The helper core strategy also showed stable behavior and allowed GPU utilization to reach 99%.
  • Bandwidth Benchmark: The overlapping strategy showed a significant performance drop (up to 85%) with large message sizes in the bandwidth benchmark, while the helper core strategy maintained a more consistent cost due to better isolation and bus utilization.

Limitations and Future Work

  • Specific Simulation Setup: The findings are specific to the tested Gromacs simulation setup (Martini model, specific hardware). The balance between CPU and GPU utilization in other simulations might differ.
  • Limited Analytics Types: The study focused on a few specific micro-benchmarks. Real-world applications might involve more complex analytics.
  • Future Work:
    • Building real-world applications combining hybrid simulations with hybrid in situ analytics.
  • Extending the study to "in-transit" analytics.
  • Investigating the impact of newer GPU features like compute capability 4.0 for prioritized kernel execution.
  • Exploring hybrid architectures with more complex node configurations.

Summary updated

BibTeX
@inproceedings{hernandez2015asynchronous,
  author = {Monica L. Hernandez and Matthieu Dreher and Carlos J. Barrios and Bruno Raffin},
  title = {{Asynchronous in Situ Processing with Gromacs: Taking Advantage of GPUs}},
  booktitle = {CARLA},
  year = {2015},
  doi = {10.1007/978-3-319-26928-3\_7},
}
Figure