Improving MPI Collective I/O Performance With Intra-node Request Aggregation

Qiao Kang, Sunwoo Lee, Kaiyuan Hou, Robert B. Ross, Ankit Agrawal, Alok N. Choudhary, Wei-keng LiaoarXiv 2019 · DOI · OpenReview

Abstract

Two-phase I/O is a well-known strategy for implementing collective MPI-IO functions. It redistributes I/O requests among the calling processes into a form that minimizes the file access costs. As modern parallel computers continue to grow into the exascale era, the communication cost of such request redistribution can quickly overwhelm collective I/O performance. This effect has been observed from parallel jobs that run on multiple compute nodes with a high count of MPI processes on each node. To reduce the communication cost, we present a new design for collective I/O by adding an extra communication layer that performs request aggregation among processes within the same compute nodes. This approach can significantly reduce inter-node communication congestion when redistributing the I/O requests. We evaluate the performance and compare with the original two-phase I/O on a Cray XC40 parallel computer with Intel KNL processors. Using I/O patterns from two large-scale production applications and an I/O benchmark, we show the performance improvement of up to 29 times when running 16384 MPI processes on 256 compute nodes.

Introduction

In high-performance computing (HPC), collective I/O is a critical mechanism that allows thousands of processes to coordinate file access. By transforming numerous small, non-contiguous writes into a smaller number of large, contiguous ones, collective I/O significantly improves efficiency. The standard implementation for this has long been the two-phase I/O strategy. In the first phase (communication), all processes send their data to a subset of "global aggregators." In the second phase (I/O), these aggregators write the consolidated data to the file system.

However, as systems scale toward the exascale era, the communication phase has become a major bottleneck. When tens of thousands of MPI processes attempt to communicate with a limited number of global aggregators simultaneously, the resulting network congestion can cause I/O performance to plummet. In many modern workloads, the time required to move data between processes now exceeds the time spent on actual disk writes.

Methods: The Two-layer Aggregation Method (TAM)

To address the "traffic jam" in the communication phase, this research introduces the Two-layer Aggregation Method (TAM). TAM inserts an additional communication layer to consolidate requests locally before they reach the global network.

The process is divided into three primary steps:

  1. Intra-node Aggregation: Processes residing on the same compute node first send their I/O requests to designated local aggregators. This step typically utilizes fast intra-node communication (e.g., shared memory), avoiding network congestion.
  2. Inter-node Aggregation: Local aggregators sort and coalesce these requests, merging adjacent data chunks into larger contiguous blocks. These consolidated requests are then sent to the global aggregators.
  3. Standard I/O Phase: The global aggregators perform the final write operations to the parallel file system (e.g., Lustre).
Hierarchical request aggregation across multiple compute nodes in the TAM framework.

By transforming the "all-to-many" communication pattern into a "many-to-few-to-fewer" hierarchy, TAM drastically reduces the fan-in degree at the global level, mitigating network contention.

Experimental Evaluation

The researchers evaluated TAM on Theta, a Cray XC40 supercomputer at Argonne National Laboratory, using up to 16,384 MPI processes. The performance was tested against three representative scientific I/O workloads:

  • E3SM-IO: Kernels from the Energy Exascale Earth System Model, focusing on "F" and "G" data decomposition cases.
  • S3D-IO: An I/O kernel from a turbulent combustion application.
  • BTIO: A block-tridiagonal I/O benchmark from the NASA NAS suite.

Key Results and Discussion

The experimental results demonstrate that network congestion, rather than raw bandwidth, is the primary bottleneck in traditional two-phase I/O. TAM addresses this by reducing the number of messages hitting global aggregators and improving data contiguity through local sorting.

  • Massive Speedups: TAM achieved speedups of up to 29x over the standard ROMIO implementation when running on 256 compute nodes (16,384 processes).
  • Performance Stability: While traditional two-phase I/O bandwidth often drops as process counts increase due to overhead, TAM maintains consistent or improved write speeds, as seen in the E3SM G-case.
  • Efficiency: The cost of the additional intra-node aggregation step was found to be negligible compared to the massive time savings gained by reducing inter-node network congestion.

Conclusion and Future Work

TAM provides a robust solution for scaling collective I/O to exascale levels. However, its performance is sensitive to parameters such as the number of local aggregators ($P_L$). Future research will focus on making the algorithm "topology-aware" to further optimize data flow based on physical network connections. Additionally, implementing pipelining to overlap the aggregation and I/O phases could provide further performance enhancements.

Summary updated

Abstract

Two-phase I/O is a well-known strategy for implementing collective MPI-IO functions. It redistributes I/O requests among the calling processes into a form that minimizes the file access costs. As modern parallel computers continue to grow into the exascale era, the communication cost of such request redistribution can quickly overwhelm collective I/O performance. This effect has been observed from parallel jobs that run on multiple compute nodes with a high count of MPI processes on each node. To reduce the communication cost, we present a new design for collective I/O by adding an extra communication layer that performs request aggregation among processes within the same compute nodes. This approach can significantly reduce inter-node communication congestion when redistributing the I/O requests. We evaluate the performance and compare with the original two-phase I/O on a Cray XC40 parallel computer with Intel KNL processors. Using I/O patterns from two large-scale production applications and an I/O benchmark, we show the performance improvement of up to 29 times when running 16384 MPI processes on 256 compute nodes.
BibTeX
@misc{kang2019improving,
  author = {Qiao Kang and Sunwoo Lee and Kaiyuan Hou and Robert B. Ross and Ankit Agrawal and Alok N. Choudhary and Wei-keng Liao},
  title = {{Improving MPI Collective I/O Performance With Intra-node Request Aggregation}},
  year = {2019},
  eprint = {1907.12656},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.12656},
}
Figure