Introduction
Modern scientific research, spanning fields from climate modeling to high-energy physics, is increasingly defined by the generation of massive datasets. Moving this data between geographically dispersed research institutions via Wide Area Networks (WAN) has become a significant bottleneck. While services like Globus have optimized the transfer process, they are ultimately constrained by physical network bandwidth. To overcome these limits, researchers have developed Ocelot, a framework that integrates error-bounded lossy compression into the Globus ecosystem, achieving transfer speedups of up to 11.2×.
The Challenge: Why Compression is Underutilized
Standard lossless compression often fails to provide meaningful gains for scientific data, which typically consists of complex, high-precision floating-point numbers. Error-bounded lossy compression offers a much higher reduction in data volume, but it introduces two primary challenges:
- Predictability: Researchers often struggle to predict how a specific error bound will affect data utility (quality) or the final file size (compression ratio).
- Computational Overhead: The time required to compress and decompress data can sometimes exceed the time saved during the transfer itself, particularly when compute resources are limited or when dealing with thousands of small files.
The Ocelot Framework
Ocelot addresses these challenges by automating the decision-making process and optimizing the execution of compression tasks. It leverages the SZ3 lossy compressor and integrates with FuncX (a Function-as-a-Service platform) and Globus to create a seamless end-to-end pipeline.
ML-Based Quality and Performance Prediction
At the heart of Ocelot is a decision tree regressor designed to estimate compression ratios, speeds, and the Peak Signal-to-Noise Ratio (PSNR). Unlike simpler models, Ocelot extracts features from three distinct categories:
- Configuration: The specific error bounds requested.
- Data-based: Entropy and value ranges of the raw data.
- Compressor-based: Internal statistics gathered during a quick initial pass, such as the distribution of quantization bins.
The researchers found that these compressor-internal features are significantly more predictive of final quality than raw data statistics alone.
Orchestration and Parallelism
Ocelot uses FuncX to trigger compression and decompression tasks remotely on high-performance computing (HPC) nodes. To ensure that computation does not become the new bottleneck, Ocelot employs MPI-based parallelization, allowing thousands of files to be processed simultaneously. Furthermore, Ocelot implements File Grouping, bundling small compressed files into larger aggregates to maximize Globus throughput and minimize I/O overhead.
The Sentinel Program: A Fail-Safe Mechanism
To solve the "node waiting time" problem, Ocelot includes a monitor known as the Sentinel Program. If compute nodes for compression are unavailable or if the predicted compression time is too high, the system automatically falls back to transferring raw data via Globus. This ensures that the total transfer time is never worse than the standard, uncompressed baseline.
Evaluation and Results
The framework was evaluated using diverse scientific datasets, including CESM (climate), Nyx (cosmology), QMCPACK (electronic structures), and Hurricane ISABEL (weather).
Key results include:
- Significant Throughput Gains: In transfers between major supercomputing sites (e.g., Purdue Anvil to NERSC Cori), Ocelot achieved speedups of up to 11.2×.
- High Prediction Accuracy: The ML model accurately predicted compression ratios and times. PSNR predictions remained within a manageable error margin (RMSE of ~13-14).
- Low Overhead: By sampling only 1% of the data for its predictions, the framework introduced an overhead of only ~1.7% of the total compression time.
- Data Integrity: Visualizations confirmed that at the high PSNR levels predicted by Ocelot, there was no visible difference between the original and reconstructed scientific data.
Limitations and Future Work
While Ocelot represents a major step forward, some limitations remain. The current ML model requires prior exposure to similar datasets to maintain high accuracy and may struggle with entirely novel data types. Additionally, while parallel decompression is efficient, it can occasionally encounter I/O contention on shared file systems. Future iterations of Ocelot will focus on auto-tuning core counts to avoid I/O bottlenecks and expanding support for transformer-based compressors like ZFP.
Conclusion
Ocelot demonstrates that by combining machine learning with high-performance orchestration, error-bounded lossy compression can be transformed from a manual, expert-only tool into an automated service. By intelligently navigating the tradeoff between computation and communication, Ocelot enables researchers to move massive scientific datasets across the globe with unprecedented efficiency.
Summary updated