Exploration of Pattern-Matching Techniques for Lossy Compression on Cosmology Simulation Data Sets

Dingwen Tao, Sheng Di, Zizhong Chen, Franck Cappello

Abstract

Exascale scientific simulations, such as the Hardware/Hybrid Accelerated Cosmology Code (HACC), generate petabytes of data that challenge modern storage and I/O infrastructures. While lossy compression is a standard mitigation strategy, the irregular nature of cosmology particle data—specifically velocity components—often results in poor performance for traditional spatial-prediction compressors. This paper introduces SZ-PM, a lossy compression algorithm that leverages dictionary-based pattern matching adapted for floating-point sequences. By utilizing sorted, mean-shifted sequences and fuzzy matching within a sliding window, the proposed method improves prediction accuracy. However, experimental evaluations reveal that the metadata overhead associated with dictionary offsets and mean values currently offsets the gains in quantization efficiency, presenting a significant hurdle for dictionary-based scientific data compression.

1. Introduction

The sheer volume of data produced by exascale simulations necessitates efficient compression. In cosmology simulations, particle data is stored as 1D arrays of coordinates and velocities. Unlike grid-based simulations, these datasets lack strong spatial correlations, making them difficult for state-of-the-art compressors like SZ and ZFP to handle effectively. As shown in Table 1, compression ratios for velocity components (vx, vy, vz) are consistently lower than those for coordinate data (xx, yy, zz), often falling between 2x and 5x.

Table 1: Baseline compression ratios for various compressors on HACC simulation data. Velocity variables (vx, vy, vz) exhibit significantly lower compressibility than coordinate variables.

2. The SZ-PM Algorithm

The authors propose SZ-PM, which builds upon the SZ lossy compression framework by incorporating a pattern-matching mechanism inspired by the LZ77 lossless algorithm. SZ-PM is specifically tailored for floating-point data and user-controlled error bounds.

The methodology consists of three primary steps:

  • Sorting and Shifting: Data points in the look-ahead buffer are sorted and shifted by their mean value to expose structural similarities (smooth curves) that are otherwise hidden in irregular data.
  • Fuzzy Matching: The algorithm searches a sliding window (Figure 1) for sequences that match the current look-ahead buffer within a specified Lp norm distance threshold.
  • Encoding: Upon finding a match, the matched sequence is used as a predictor. The encoder records a match flag, the buffer offset, the sequence mean, and the quantized residuals. If no match is found, the system falls back to the standard SZ prediction model.
Figure 1: Diagram of the sliding window architecture, illustrating the relationship between the search buffer and the look-ahead buffer in dictionary-based encoding.

3. Experimental Evaluation

The performance of SZ-PM was evaluated using HACC velocity data (vx) with a relative error bound of $10^{-4}$. The results demonstrate that the pattern-matching approach significantly improves quantization. For a sequence length of 8, the quantization code size was reduced from 7.31 bits/value (standard SZ) to 5.45 bits/value.

Despite this improvement, the overall bit-rate increased due to metadata overhead. Storing the dictionary offset added 1.25 bits/value, while the sequence mean added 3.98 bits/value. Consequently, the compression ratio for SZ-PM at a sequence length of 8 was 2.96, failing to surpass the baseline SZ ratio of 4.38. Even with a sequence length of 32, SZ-PM achieved a ratio of 4.71, which remained below the baseline of 5.19.

4. Conclusion and Future Work

This research highlights a fundamental trade-off in applying dictionary-based techniques to scientific data: while pattern matching enhances prediction accuracy for irregular datasets, the resulting metadata can negate compression gains. Furthermore, the sorting process requires reordering corresponding coordinate arrays to maintain consistency, which may impact overall dataset compressibility.

Future efforts will focus on optimizing metadata storage, such as using alternative reference values (e.g., the first element of a sequence) to eliminate the need for storing sequence means. Exploring more advanced pattern-matching techniques remains a priority to further boost prediction accuracy and achieve net positive compression ratios.

Summary updated

BibTeX
@inproceedings{tao2017exploration,
  author = {Dingwen Tao and Sheng Di and Zizhong Chen and Franck Cappello},
  title = {{Exploration of Pattern-Matching Techniques for Lossy Compression on Cosmology Simulation Data Sets}},
  booktitle = {ISC Workshops},
  year = {2017},
  doi = {10.1007/978-3-319-67630-2\_4},
}
Figure