Abstract
Atmospheric states derived from reanalysis comprise a substantial portion of weather and climate simulation outputs. Many stakeholders -- such as researchers, policy makers, and insurers -- use this data to better understand the earth system and guide policy decisions. Atmospheric states have also received increased interest as machine learning approaches to weather prediction have shown promising results. A key issue for all audiences is that dense time series of these high-dimensional states comprise an enormous amount of data, precluding all but the most well resourced groups from accessing and using historical data and future projections. To address this problem, we propose a method for compressing atmospheric states using methods from the neural network literature, adapting spherical data to processing by conventional neural architectures through the use of the area-preserving HEALPix projection. We investigate two model classes for building neural compressors: the hyperprior model from the neural image compression literature and recent vector-quantised models. We show that both families of models satisfy the desiderata of small average error, a small number of high-error reconstructed pixels, faithful reproduction of extreme events such as hurricanes and heatwaves, preservation of the spectral power distribution across spatial scales. We demonstrate compression ratios in excess of 1000x, with compression and decompression at a rate of approximately one second per global atmospheric state.
Introduction: The Atmospheric Data Deluge
The scientific community is currently facing a "data deluge" in climate and weather modeling. The ECMWF Reanalysis v5 (ERA5), the gold standard for historical atmospheric data, requires approximately 181 terabytes of storage for a single variable over a 40-year span. When considering multiple variables, vertical levels, and ensemble trajectories, the data volume quickly escalates into the petabyte range.
This massive storage requirement creates a significant barrier to entry, limiting high-resolution climate research to only the most well-resourced institutions. Standard lossless compression techniques, such as ZIP, typically yield only a 2x reduction in size. Conversely, traditional lossy compression methods like JPEG often introduce artifacts that blur high-frequency details, effectively erasing small-scale but high-impact phenomena like storms. There is a critical need for a compression method that can achieve three orders of magnitude reduction without compromising the underlying physical integrity of the data.
Methodology
The proposed neural compression pipeline treats atmospheric states as data on a sphere rather than simple flat images. The methodology consists of four primary stages:
- HEALPix Projection: To avoid the distortion and oversampling of poles inherent in standard latitude/longitude grids, the global data is projected onto 12 equal-area HEALPix squares. This projection allows the data to be processed efficiently by standard neural network architectures.
- Neural Autoencoders: The study evaluates two main architectures: the VQ-VAE/VQ-GAN, which maps data into a discrete "codebook" of representative patterns, and the Hyperprior Model. The latter, adapted from state-of-the-art image compression, utilizes a secondary "side" network to learn the probability distribution of the compressed data, enabling more efficient entropy coding.
- Physical Conditioning: The models are conditioned on "orography" data, including log-elevation and land/sea masks, to provide the network with essential geographical context.
- Spectral Preservation: Reconstructions utilize spherical harmonic transforms to ensure that global spectral properties—the distribution of energy across different spatial scales—are maintained.
Experimental Results
The models were trained on 51 years of ERA5 data (1959–2009) and tested on data from 2010–2023. The results demonstrate that neural compression can achieve a 1000:1 compression ratio while maintaining remarkable accuracy.
Reconstruction Fidelity and Error Analysis
The hyperprior model significantly outperformed the VQ-VAE approach by learning to predict the variance of compressed bits. Quantitative analysis shows a Mean Absolute Error (MAE) of approximately 0.4°K for temperature and 0.5 m/s for wind components. Crucially, high-error pixels are rare, occurring in less than 0.5% of the data. The researchers suggest these rare "bad pixels" could even be stored in a tiny separate lookup table to guarantee near-perfect reconstruction of extreme events.
As shown in the multi-variable evaluation, the visual fidelity of the reconstructions is high. In regional views of the Caribbean, the model accurately preserves the structure of intense weather systems, such as the eye and wind fields of Hurricane Matthew. Unlike baseline methods that often "wash out" such features, the neural compressor maintains sharp gradients and physical consistency. Furthermore, the compressor helped identify artifacts in the original ERA5 data, such as physically impossible negative humidity values, which appeared as spikes in reconstruction error.
Computational Efficiency
Compression and decompression are highly efficient, taking approximately one second per global state. This speed makes the method viable for real-time data serving and large-scale historical analysis on standard hardware.
Discussion and Future Work
While the results are promising, several limitations remain. Unlike traditional error-bounded compressors (e.g., SZ3), neural networks do not yet provide a hard mathematical guarantee on maximum error, which may be necessary for certain high-precision applications. Additionally, "joint variable contamination" can occur, where artifacts in one compressed variable might slightly affect another due to shared latent representations.
Future research will address the stationarity of these models. As climate change shifts the Earth's baseline state further from the historical training data, models may require periodic retraining to maintain accuracy over long-term projections. Nevertheless, this work demonstrates that "cloud-scale" petabytes of climate data can be reduced to "desktop-scale" terabytes, democratizing access to vital earth system information.
Summary updated