PointZip: Statistically Lossless Compression for Climate Data with Pointwise Error Bounds

Jiayong Li, Pratyai Mazumder, Yuejiang Yu, Langwen Huang, Alexandru Calotoiu, Torsten HoeflerarXiv 2026 · arXiv

TL;DR

PointZip is a new "statistically lossless" compression framework for climate data that bounds compression errors using the data's own intrinsic uncertainty (ensemble spread). It achieves massive compression ratios—up to 72x for high-resolution simulations—effectively breaking the I/O bottleneck that currently prevents researchers from saving and analyzing 1km-scale climate models.

What Problem Does This Paper Address?

High-resolution climate simulations, like the ICON model, are reaching a breaking point. While we have the compute power to simulate the Earth at 1km resolution, we don't have the storage bandwidth to save the results. In some cases, the time required to write a single simulation timestep to disk is longer than the time it took to compute it.

Existing solutions fall short: purely lossless compression only offers a modest ~2x reduction, while "lossy" compression requires users to manually set error tolerances. Setting these tolerances is a scientific minefield—too loose and you ruin the data's physical validity; too tight and you don't save enough space. PointZip solves this by defining "statistically lossless" compression: if the compression error at any given point is smaller than the inherent uncertainty (noise) already present in the data, the compression is scientifically indistinguishable from the original.

Methods

PointZip uses a two-stage architecture to enforce pointwise error bounds while maximizing efficiency:

  1. Base Layer: It first applies a standard, rate-distortion-efficient spatial codec (currently JPEG2000). This captures the broad spatial structures of the climate data.
  2. Residual Coding: Since the base layer doesn't guarantee pointwise accuracy, PointZip calculates the "residual" (the difference between the original and the base-layer version). It then applies Error-Bound-Aware Precision Truncation.

As shown in Figure 2, if the residual is already within the uncertainty bound, it is masked to zero. If it exceeds the bound, it is rounded to the coarsest possible value that still satisfies the bound. These truncated residuals are then stored losslessly using sparse coding (bitmaps and ZSTD).

![Fig.

  1. Precision truncation process. The truncation error is never larger than the error bound.](/figures/art_9f96bda2d04543e49d73968e9c330e27)

The system also includes an optional Feedback Optimization loop (Figure 1) that can automatically tune the base layer's compression level to find the "sweet spot" that minimizes the total size of both layers combined.

![Fig.

  1. Overview of PointZip pipeline.](/figures/art_d1e030645d464386916ec990bbc92a6e)

Datasets

The authors evaluated PointZip on two primary targets:

  • ERA5: The global climate reanalysis dataset. They used the 10-member ensemble spread to define the pointwise uncertainty bounds.
  • ICON: Output from a state-of-the-art 1km resolution simulation.
  • Downstream Benchmarks: To ensure scientific validity, they tested the compressed data on neural weather forecasting (Aurora model), Lagrangian air parcel trajectories (LAGRANTO), and spectral analysis.

Key Insights and Ideas

  • Uncertainty as a Bound: The most elegant idea here is moving away from arbitrary percentage-based error bounds. By using the ensemble spread (the standard deviation across multiple simulation runs), the error bound becomes a physically grounded property of the dataset itself.
  • Breaking the I/O Wall: For the ICON model, PointZip reduced the I/O time for a single timestep from 82.2 seconds to just 3.33 seconds. This transforms high-res simulations from "impossible to store" to "practical for daily use."
  • Preserving the "Highs": Standard lossy compressors often smooth out sharp gradients (high-frequency data). Because PointZip's residual layer specifically targets points where the base layer failed, it does an exceptional job at preserving extreme events like hurricanes and fine-scale atmospheric spectra.

Key Results

  • Massive Compression: PointZip achieved a 69.3x compression ratio on representative ERA5 variables. This allows 40 years of the most commonly used climate data (275 TB) to be compressed down to just 4 TB—fitting on a single consumer hard drive.
  • High Throughput: The system is fast. Multi-threaded implementation on a single node reached 4.3 GB/s, which exceeds the data production rate of major centers like the ECMWF.
  • Scientific Fidelity: In neural weather forecasting tests, the Root Mean Square Error (RMSE) increased by less than 1%. In air parcel trajectory simulations, PointZip maintained high accuracy (FSS > 0.8) even after 7 days of simulated transport, significantly outperforming other lossy compressors.

![Fig.

  1. Throughput and compression ratio comparison between PointZip and EBCC.](/figures/art_d33a4b3a2a9340338074ad8a8f143549)

Limitations and Future Work

  • Dependency on Uncertainty Estimates: PointZip requires an uncertainty field (like an ensemble spread) to function in its "statistically lossless" mode. While common in modern reanalysis, not all legacy datasets have this information.
  • Floating-Point Mapping: The current use of JPEG2000 requires mapping floating-point data to 16-bit integers. While the authors show this introduces negligible error compared to the uncertainty bounds, a native floating-point base codec might be more elegant.
  • Future Directions: The authors suggest that PointZip could be integrated directly into simulation runtimes to enable "on-the-fly" compression, further reducing the memory and bandwidth pressure on supercomputing nodes.

Summary updated

Abstract

High-resolution climate simulations generate data volumes that increasingly strain storage, I/O, and data dissemination. Error-bounded lossy compressors can reduce these costs, but their tolerances are usually user-chosen and difficult to standardize across heterogeneous climate datasets. We address this limitation with statistically lossless compression, where the pointwise compression error is bounded by the pointwise data uncertainty. We propose PointZip, a two-stage compressor that combines a rate-distortion-efficient base layer with error-bound-aware residual coding to enforce pointwise guarantees. On outputs from the widely used climate modeling framework ICON, PointZip makes 1km simulations practical by reducing storage bandwidth requirements by 72x. On the ERA5 climate dataset, PointZip reaches 69.3x on the most representative variables, reducing the commonly used subset to 4 TB. In downstream evaluations, PointZip preserves data spectra, extreme events, and air parcel trajectories, while increasing neural weather forecast RMSE by less than 1%, with negligible impact on the Fractions Skill Score (FSS).
BibTeX
@misc{li2026pointzip,
  author = {Jiayong Li and Pratyai Mazumder and Yuejiang Yu and Langwen Huang and Alexandru Calotoiu and Torsten Hoefler},
  title = {{PointZip: Statistically Lossless Compression for Climate Data with Pointwise Error Bounds}},
  year = {2026},
  eprint = {2611.21312},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21312},
}
Figure