Scientific simulations in the exascale era generate massive datasets that frequently overwhelm storage capacity and I/O bandwidth. For instance, a 1km resolution climate simulation can produce 260 TB of floating-point data every 16 seconds. While lossless compression like GZIP typically offers only a 2x reduction for scientific data, traditional lossy compression (e.g., JPEG) lacks the strict error guarantees required for scientific integrity. Error-bounded lossy compression has emerged as a critical solution, ensuring decompressed data remains within user-defined tolerances.
However, existing state-of-the-art compressors like SZ2 or ZFP are often monolithic. Because scientific datasets vary wildly in mathematical characteristics—from quantum chemistry to climate modeling—a one-size-fits-all approach is suboptimal. SZ3 addresses this by introducing a highly modular framework that allows researchers to compose custom compression pipelines tailored to specific data characteristics.
The SZ3 Modular Architecture
SZ3 decomposes prediction-based compression into five independent, interchangeable modules:
- Preprocessor: Transforms data (e.g., logarithmic transforms) before the core compression steps.
- Predictor: Estimates data values based on neighbors (e.g., Lorenzo or linear regression).
- Quantizer: Maps prediction errors to discrete integers while respecting error bounds.
- Encoder: Performs lossless reduction of quantized values (e.g., Huffman encoding).
- Lossless Compressor: A final pass (e.g., ZSTD) to eliminate remaining redundancies.
To maintain high performance, SZ3 utilizes C++ compile-time polymorphism (templates) instead of runtime virtual functions. This design ensures that modularity does not introduce significant computational overhead. Furthermore, the introduction of a multidimensional iterator and datatype abstraction allows SZ3 to support 1D through 4D data seamlessly while maintaining a significantly smaller codebase than its predecessor, SZ2.
Customizing Pipelines: SZ3-Pastri
The power of SZ3 is best demonstrated through specialized pipelines. For example, the SZ3-Pastri pipeline was developed for quantum chemistry data (GAMESS). It introduces an "Unpred-aware Quantizer" that uses bitplane encoding to efficiently handle unpredictable data outliers that typically degrade compression ratios in standard predictors.
Another example is SZ3-APS, an adaptive pipeline for X-ray ptychography. It dynamically switches between 1D and 3D predictors depending on the user's requested error bound, optimizing for the specific mathematical properties of the data at different precision levels.
Key Insights
- No Universal Compressor: The optimal compression strategy varies not only by dataset but also by the requested error bound. SZ3’s adaptive pipelines can manage these transitions automatically, ensuring the best performance across a wide range of scenarios.
- Handling Unpredictability: Data points falling outside a predictor's expected range often ruin compression ratios. SZ3 allows for specialized quantizers that salvage these outliers, which is crucial for complex datasets like those found in quantum chemistry.
- Modularity without Overhead: A common concern in software engineering is that abstraction leads to performance loss. SZ3 proves that with modern C++ techniques, a modular framework can actually outperform tightly coupled legacy code.
Performance Results
Evaluations across diverse scientific benchmarks—including Cosmology (HACC, NYX), Climate (ATM, Hurricane), and Turbulence (Miranda)—highlight SZ3's advantages:
- Compression Ratio: On GAMESS data, the SZ3-Pastri pipeline improved compression ratios by 20% to 40% over previous state-of-the-art methods at strict error bounds (e.g., 1E-10).
- Adaptivity: The SZ3-APS pipeline achieved up to an 18% gain in compression ratio for X-ray data by dynamically selecting predictors.
- General Performance: The interpolation-based pipeline (
SZ3-Interp) achieved a 56% higher compression ratio than classic regression pipelines on turbulence datasets. - Throughput: For speed-critical applications, the
SZ3-Truncationpipeline reached throughputs exceeding 1 GB/s, outperforming competitors by 4x.
Conclusion and Future Work
While highly specialized pipelines can sometimes trade compression speed for higher ratios, SZ3 provides the flexibility to choose the right balance for any given workload. The authors note that the complex encoding stages in pipelines like SZ3-Pastri can reduce computational throughput, but the storage savings often justify the cost.
Future developments will focus on expanding the library of module instances to cover a wider array of scientific use cases. Crucially, the team plans to extend SZ3 to support hardware accelerators, including GPUs and FPGAs, which will be vital for the next generation of exascale supercomputers.
Summary updated