Unbalanced Parallel I/O: An Often-Neglected Side Effect of Lossy Scientific Data Compression

Xinying Wang, Lipeng Wan, Jieyang Chen, Qian Gong, Ben Whitney, Jinzhen Wang, Ana Gainaru, Qing Liu, Norbert Podhorszki, Dongfang Zhao, Feng Yan, Scott KlaskyDRBSD@SC 2021 · DOI

Introduction

As we approach the exascale era, high-performance computing (HPC) simulations are generating data at an unprecedented rate. Simulations like the X-point Gyrokinetic Code (XGC) can produce over a petabyte of data daily, necessitating the use of sophisticated lossy compressors such as MGARD, ZFP, and SZ to manage storage and transfer overhead. While these tools effectively reduce data volume while preserving scientific integrity, they introduce a significant, often-overlooked side effect: I/O imbalance.

A recent study by Xinying Wang and colleagues (ORNL, UNR, and NJIT) highlights how lossy compression creates performance bottlenecks by inducing highly skewed data sizes across parallel computing processes. This "straggler effect" prevents researchers from achieving the full performance gains expected from data reduction.

The Mechanics of I/O Imbalance

In traditional HPC workflows, "domain decomposition" is used to split a simulation into equal-sized spatial chunks, each assigned to a specific MPI process. This typically ensures balanced I/O, as every process writes an identical amount of data simultaneously.

However, lossy compressors are content-aware; they achieve higher compression ratios in regions of low information density (low entropy) and lower ratios in regions with complex physics (high entropy). When a simulation features localized phenomena—such as particle clustering or turbulent eddies—the processes handling those regions produce significantly larger compressed outputs than their neighbors.

Spatial variation in simulation data leads to non-uniform information density across the domain, which directly impacts local compression ratios.

Characterizing the Skew

The researchers analyzed three major scientific applications: the Gray-Scott reaction-diffusion model, the WarpX electromagnetic Particle-In-Cell (PIC) code, and the XGC fusion plasma simulation. Their findings revealed that the distribution of compressed data sizes across processes closely follows a Weibull distribution.

In the WarpX simulation, the imbalance was extreme: some processes produced compressed data hundreds of times larger than others. Even in more uniform simulations, a small subset of processes (roughly 5%) often handled data loads 10x larger than the average. This creates a bottleneck where the entire parallel system must wait for the slowest "straggler" to finish its I/O operations.

Histogram of compressed data sizes for the Gray-Scott simulation using ZFP, illustrating the significant skewness where a majority of processes have small loads while a few "stragglers" have much larger ones.

Performance Impact and Results

To measure the real-world impact, the team conducted benchmarks on the Andes cluster at Oak Ridge National Laboratory. They found that the theoretical benefits of compression are severely undercut by this imbalance:

  • Diminishing Returns: A 100x reduction in total data volume rarely translates to a 100x speedup. In practice, the authors observed gains as low as 6x due to the straggler effect.
  • Sensitivity of Shared Files: Writing to a single shared file (N-1 I/O) proved significantly more sensitive to load imbalance than writing to individual per-process files (N-N I/O).
  • The Clustered Bottleneck: The worst performance occurred when high-density data chunks were physically located on the same compute node, causing them to compete for the same local bandwidth and further slowing the system.

Conclusion and Future Directions

While lossy compression is essential for exascale science, current I/O libraries are not optimized for the dynamic imbalance it creates. The study suggests several paths forward to mitigate these bottlenecks, including predictive load balancing based on information theory, "smart" process mapping to pair high-density and low-density chunks on the same node, and the development of dynamic I/O scheduling within libraries like ADIOS2 or MPI-IO. Addressing this hidden cost is vital for ensuring that compression remains a viable solution for the next generation of supercomputing.

Summary updated

BibTeX
@inproceedings{wang2021unbalanced,
  author = {Xinying Wang and Lipeng Wan and Jieyang Chen and Qian Gong and Ben Whitney and Jinzhen Wang and Ana Gainaru and Qing Liu and Norbert Podhorszki and Dongfang Zhao and Feng Yan and Scott Klasky},
  title = {{Unbalanced Parallel I/O: An Often-Neglected Side Effect of Lossy Scientific Data Compression}},
  booktitle = {DRBSD@SC},
  year = {2021},
  doi = {10.1109/drbsd754563.2021.00008},
}
Figure