Error Bounded Lossy Compression for Scientific Simulations

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2223 papers249 conceptsfrozen 2026-10-04

Modern supercomputers have a fundamental imbalance: they can calculate data much faster than they can write it to disk. As we move toward exascale computing, simulations of climate systems, turbulent flows, and cosmic structures generate petabytes of data that simply cannot be stored or shared in their raw form.

Traditional lossless compression, which guarantees bit-for-bit reconstruction, typically fails to exceed a 2:1 reduction ratio for scientific floating-point data. To break this bottleneck, researchers are developing error-bounded lossy compression—a suite of techniques that aggressively shrink data by discarding "insignificant" information while providing mathematical guarantees that the scientific integrity of the data remains intact.

The Exascale I/O Bottleneck

In high-performance computing (HPC), the time spent on Input/Output (I/O) often stalls the actual physics calculations. As Sheng Di, Franck Cappello, and colleagues note in A Survey on Error-Bounded Lossy Compression for Scientific Datasets, the data volume from kilometer-scale atmospheric models or trillion-particle cosmological simulations creates a "sustainability crisis."

If a researcher cannot save the full-resolution volumetric data for every timestep, they are forced to either downsample the data—losing critical features—or find a way to represent the data more efficiently. The goal of error-bounded compression is to ensure that the difference between the original and reconstructed data stays within a user-defined tolerance (e.g., a pointwise relative error of $10^{-4}$), ensuring that downstream analysis like turbulence statistics or climate trends remains valid.

Prediction and Quantization: The SZ and ZFP Paradigms

The two dominant "classical" approaches to this problem are SZ and ZFP.

  • SZ uses a prediction-based approach. It looks at neighboring data points to predict the next value and then only stores the "quantized" error of that prediction. If the prediction is accurate, the error is small and can be represented with very few bits.
  • ZFP uses a transform-based approach, similar to JPEG but for multidimensional scientific data. It breaks data into small blocks and applies a custom orthogonal transform to concentrate the energy into a few coefficients.

Choosing between these is often difficult. Sheng Di and Xin Liang developed an online selection algorithm that automatically picks the best compressor for a specific dataset on-the-fly, improving compression ratios by up to 70%.

Neural Representations: Data as Network Weights

A more recent and radical shift involves treating scientific data not as a grid of numbers, but as a continuous function. By "overfitting" a neural network to a specific simulation, the network weights themselves become the compressed representation of the data.

Torsten Hoefler demonstrated that overfitting climate data into coordinate-based neural networks can achieve compression ratios between 300x and 3,000x. Similarly, Sandeep Suresh Cranganore and Johannes Brandstetter introduced Einstein Fields, which compresses 4D numerical relativity simulations into network weights.

The intuition here is that physical fields (like gravity or wind velocity) have immense internal structure governed by PDEs. Neural networks are excellent at capturing these smooth, continuous patterns. This approach not only saves space but also allows for "free" spatial interpolation—you can query the network for a value at any coordinate, not just the original grid points.

Physics-Informed Compression

To ensure that compressed data doesn't violate the laws of physics, researchers are integrating physical constraints directly into the compression pipeline. For example, Enmao Diao and Andrew D. Bragg developed a Physics-Informed Vector-Quantized Autoencoder (VQ-AE) for turbulent flows. By enforcing incompressibility (divergence-free fields) during the training of the autoencoder, they achieved an 85:1 compression ratio while preserving the small-scale statistical features essential for fluid dynamics research.

In-Situ Processing and Super-Resolution

Because the bottleneck is the disk, the most efficient place to compress data is in-situ—while it is still in the supercomputer's memory.

Han Wei Shen and colleagues have explored hierarchical super-resolution as a compression strategy. Instead of saving a high-resolution volume, the simulation saves a low-resolution version. Later, a deep learning model upscales the data, recovering the lost details. To handle the uncertainty inherent in this process, PSRFlow uses flow-based generative models to provide probabilistic reconstructions, telling scientists not just what the missing data might look like, but how confident the model is in that reconstruction.

The Path Forward

The research direction is clear: as simulations grow, we must move away from storing raw bits and toward storing compact, intelligent representations. Whether through parallel tensor decompositions or transformer-based language models for data, the goal is to transform the exascale data deluge into a manageable, interactive resource for scientific discovery.

Papers in this version

Showing 60 of 223 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 109 papers
  2. Version 1 · 2026-10-04 · 223 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_scientific_data_storage_bottleneck_ccdfb92d,
  title = {scientific data storage bottleneck},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scientific-data-storage-bottleneck/rdr_ccdfb92d025043079bdffd06d831b7ac},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_error_bounded_lossy_compression_for_scie_ccdfb92d_v1,
  title = {Error Bounded Lossy Compression for Scientific Simulations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/error-bounded-lossy-compression-for-scientific-simulations/rdr_ccdfb92d025043079bdffd06d831b7ac/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}