SZ3: A Modular Framework for Composing Prediction-Based Error-Bounded Lossy Compressors

Xin Liang, Kai Zhao, Sheng Di, Sihuan Li, Robert Underwood, Ali Murat Gok, Jiannan Tian, Junjing Deng, Jon C. Calhoun, Dingwen Tao, Zizhong Chen, Franck Cappello

Scientific simulations in the exascale era generate massive datasets that frequently overwhelm storage capacity and I/O bandwidth. For instance, a 1km resolution climate simulation can produce 260 TB of floating-point data every 16 seconds. While lossless compression like GZIP typically offers only a 2x reduction for scientific data, traditional lossy compression (e.g., JPEG) lacks the strict error guarantees required for scientific integrity. Error-bounded lossy compression has emerged as a critical solution, ensuring decompressed data remains within user-defined tolerances.

However, existing state-of-the-art compressors like SZ2 or ZFP are often monolithic. Because scientific datasets vary wildly in mathematical characteristics—from quantum chemistry to climate modeling—a one-size-fits-all approach is suboptimal. SZ3 addresses this by introducing a highly modular framework that allows researchers to compose custom compression pipelines tailored to specific data characteristics.

The SZ3 Modular Architecture

SZ3 decomposes prediction-based compression into five independent, interchangeable modules:

  1. Preprocessor: Transforms data (e.g., logarithmic transforms) before the core compression steps.
  2. Predictor: Estimates data values based on neighbors (e.g., Lorenzo or linear regression).
  3. Quantizer: Maps prediction errors to discrete integers while respecting error bounds.
  4. Encoder: Performs lossless reduction of quantized values (e.g., Huffman encoding).
  5. Lossless Compressor: A final pass (e.g., ZSTD) to eliminate remaining redundancies.
SZ3 Modular Framework Abstraction and Instances

To maintain high performance, SZ3 utilizes C++ compile-time polymorphism (templates) instead of runtime virtual functions. This design ensures that modularity does not introduce significant computational overhead. Furthermore, the introduction of a multidimensional iterator and datatype abstraction allows SZ3 to support 1D through 4D data seamlessly while maintaining a significantly smaller codebase than its predecessor, SZ2.

Customizing Pipelines: SZ3-Pastri

The power of SZ3 is best demonstrated through specialized pipelines. For example, the SZ3-Pastri pipeline was developed for quantum chemistry data (GAMESS). It introduces an "Unpred-aware Quantizer" that uses bitplane encoding to efficiently handle unpredictable data outliers that typically degrade compression ratios in standard predictors.

The SZ3-Pastri Pipeline for Quantum Chemistry Data

Another example is SZ3-APS, an adaptive pipeline for X-ray ptychography. It dynamically switches between 1D and 3D predictors depending on the user's requested error bound, optimizing for the specific mathematical properties of the data at different precision levels.

Key Insights

  • No Universal Compressor: The optimal compression strategy varies not only by dataset but also by the requested error bound. SZ3’s adaptive pipelines can manage these transitions automatically, ensuring the best performance across a wide range of scenarios.
  • Handling Unpredictability: Data points falling outside a predictor's expected range often ruin compression ratios. SZ3 allows for specialized quantizers that salvage these outliers, which is crucial for complex datasets like those found in quantum chemistry.
  • Modularity without Overhead: A common concern in software engineering is that abstraction leads to performance loss. SZ3 proves that with modern C++ techniques, a modular framework can actually outperform tightly coupled legacy code.

Performance Results

Evaluations across diverse scientific benchmarks—including Cosmology (HACC, NYX), Climate (ATM, Hurricane), and Turbulence (Miranda)—highlight SZ3's advantages:

  • Compression Ratio: On GAMESS data, the SZ3-Pastri pipeline improved compression ratios by 20% to 40% over previous state-of-the-art methods at strict error bounds (e.g., 1E-10).
  • Adaptivity: The SZ3-APS pipeline achieved up to an 18% gain in compression ratio for X-ray data by dynamically selecting predictors.
  • General Performance: The interpolation-based pipeline (SZ3-Interp) achieved a 56% higher compression ratio than classic regression pipelines on turbulence datasets.
  • Throughput: For speed-critical applications, the SZ3-Truncation pipeline reached throughputs exceeding 1 GB/s, outperforming competitors by 4x.

Conclusion and Future Work

While highly specialized pipelines can sometimes trade compression speed for higher ratios, SZ3 provides the flexibility to choose the right balance for any given workload. The authors note that the complex encoding stages in pipelines like SZ3-Pastri can reduce computational throughput, but the storage savings often justify the cost.

Future developments will focus on expanding the library of module instances to cover a wider array of scientific use cases. Crucially, the team plans to extend SZ3 to support hardware accelerators, including GPUs and FPGAs, which will be vital for the next generation of exascale supercomputers.

Summary updated

BibTeX
@article{liang2023sz,
  author = {Xin Liang and Kai Zhao and Sheng Di and Sihuan Li and Robert Underwood and Ali Murat Gok and Jiannan Tian and Junjing Deng and Jon C. Calhoun and Dingwen Tao and Zizhong Chen and Franck Cappello},
  title = {{SZ3: A Modular Framework for Composing Prediction-Based Error-Bounded Lossy Compressors}},
  journal = {IEEE Trans. Big Data},
  year = {2023},
  doi = {10.1109/tbdata.2022.3201176},
}
Figure