Kilometer-scale weather and climate models produce more data than supercomputers can write to disk. They are therefore compressed with lossy methods, which keep each value only up to a chosen error, and nobody knows how far those methods sit from the best possible. The obvious yardstick, Shannon's rate-distortion function (the minimum bits per sample needed to reproduce data within a given average error), assumes the data's statistics settle down over long records. Convective weather does not settle down in this way. Its variance and correlations swing with the solar day. They also drift against the calendar, because the year is not a whole number of days. Storm blocks need many more bits than quiet ones. A recent line of information theory, built for communication signals sampled by clocks that drift against the signal's own period, computes compression limits exactly in this regime. It approximates the drifting case by a sequence of perfectly aligned cases and takes their limit superior, the largest value the sequence keeps returning to.
We propose to carry that machinery into atmospheric data and to use it for a concrete systems question. The question is what peak bit rate a bounded-delay output stream from a convection-permitting model must provision so that only a fraction $\epsilon$ of data blocks, typically the storm blocks, ever exceed their error tolerance. The answer is a quantile of the information spectrum, the distribution of per-block information content, not its mean. We will estimate it from data, test whether it predicts where real compressors spend their bits, and check whether a much simpler phase-normalization recipe gives the same answer.
Core Claim
Hourly or sub-daily atmospheric fields at a fixed location are, to second order, periodically varying processes. Their period is 24 hours, and a slower annual cycle sits on top that the day cannot divide evenly. Any sampling or blocking cadence that is commensurate with one cycle is incommensurate with the other. Non-sun-synchronous satellite overpasses and adaptive output schedules add further asynchrony. Such data fall into the class the communication literature calls almost cyclostationary: their statistics repeat only approximately, so sample averages of information content need not concentrate.
The claim has three parts. The first is that the bounded-delay characterization of The Rate-Distortion Function for Sampled Cyclostationary Gaussian Processes with Memory and with Bounded Processing Delay can be turned into a data-driven estimator. The estimator takes cyclic covariances estimated from reanalysis or model output and returns a computable rate-distortion curve. Because a Gaussian source is the hardest to compress among all sources with the same covariance under squared error, this curve is a valid upper bound on the bits an optimal coder would need, even for non-Gaussian precipitation. The second part is that the per-block spread of rates, a plug-in estimate of the information spectrum, has an upper tail that coincides with convective extremes. That tail sets the provisioning rate for streaming with bounded buffer delay. The third part is a falsifiable prediction. Atmospheric fields have long temporal memory, so the extreme sensitivity to cadence seen in communications should be strongly damped here, while the choice of where block boundaries fall in the solar day should still matter at finite block lengths.
What Prior Work Supplies and What It Leaves Open
The method comes from a tightly connected group of papers. The Distortion Rate Function of Cyclostationary Gaussian Processes derives compression limits for periodically varying Gaussian signals. It decomposes them into phase components and allocates bits by reverse waterfilling, which spends bits only on spectral modes whose power exceeds a threshold. On the Rate-Distortion Function of Sampled Cyclostationary Gaussian Processes handles the drifting-clock case for memoryless sources. It sandwiches the asynchronous limit between synchronous approximations and takes a limit superior, but only in the low-distortion regime. On the Rate-Distortion Function for Sampled Cyclostationary Gaussian Processes with Memory adds temporal memory, at the cost of a formula with an infinite-block optimization that cannot be computed. The bounded-delay paper restores computability by inserting gaps between blocks and aligning blocks to an optimal starting phase. That construction is the one we reuse.
The Capacity of Memoryless Channels With Sampled Cyclostationary Gaussian Noise motivates the cadence experiment. It reports that a 0.2% change in sampling rate can shift capacity by up to 30%, and that capacity becomes invariant to time offsets once sampling is truly asynchronous. All four papers are confined to synthetic Gaussian signals. None estimates cyclic statistics from physical data. None looks at the distribution of per-block rates, only the asymptotic rate.
The information-spectrum side, with inf- and sup-rates defined as probabilistic lower and upper limits of normalized information density, is supplied by the general formulas in A Formula for the Capacity of the General Gel'fand-Pinsker Channel and the $\epsilon$-rate bounds of A Lower and Upper Bound on the Epsilon-Uniform Common Randomness Capacity. We borrow only the idea that a quantile of the information density, rather than its mean, governs rates when a nonzero failure fraction $\epsilon$ is allowed. Neither paper touches lossy compression of physical fields.
The atmospheric papers supply both the need and the constraints. Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 GPUs with COSMO 5.0 documents throughput of 0.043 simulated years per day at 930 m and the resulting data deluge. Compressing multidimensional weather and climate data into neural networks shows that coordinate-based neural networks reach 300× to 3,000× compression on ERA5 and beat the scientific compressor SZ3 above 400×. It also shows that they smooth out extremes such as hurricane pressure minima and need about 8 hours of GPU encoding. That is a direct constraint: high average compression is not the same as preserving storms. Diagnosing added value of convection-permitting regional models using precipitation event identification and tracking shows that explicit convection changes the diurnal phase and size of rain events. This is why the cyclic structure we model is resolution-dependent and must be estimated, not assumed. HR-Extreme: A High-Resolution Dataset for Extreme Weather Forecasting provides labeled 3 km extreme-event boxes. These boxes let us check whether the upper tail of the information spectrum lines up with physically identified hazards.
Research Program
The first component is an estimator. For each grid column, and for the leading spatial principal components of regional patches, we estimate the cyclic mean and the cyclic autocovariance as functions of local solar time and day of year. We use multi-decade hourly ERA5 and the hourly 3 km HRRR fields packaged in HR-Extreme. From these estimates we build the synchronous approximations prescribed by the limit-superior construction. This means rational approximations to the ratio of block cadence to the day-and-year cycle, with increasing common period, each solved by phase decomposition and reverse waterfilling under a bounded inter-block delay. We report the limit and its convergence. Precipitation and other skewed variables are run both raw and after a Gaussianizing transform, a monotone map to normal scores. The two runs separate the effect of marginal non-Gaussianity from that of cyclic correlation.
The second component is the spectrum. We compute the reverse-waterfilling rate of each finite block, conditioned on that block's phase and its own estimated covariance. Collected over the archive, these values form an empirical distribution that serves as a plug-in estimate of the information spectrum. Its $\epsilon$-quantile is our candidate provisioning rate. We state plainly that this is a model-based approximation of the information spectrum, not the true spectrum of the data.
The third component is an audit. We run real compressors on the same blocks and compare their rates, total and per-block, with these bounds. We vary block cadence and the solar-time phase at which blocks start.
Baselines
The naive reference point is the stationary Gaussian bound. It fits one covariance to the whole record, ignores the cycle, and computes rate-distortion by ordinary waterfilling, under the same blocks, the same distortion targets and the same delay. Beating it means the cyclic bound is materially tighter at convective sites. We would expect at least several percent of rate at matched error over continental summer regimes, and near zero gap over high-latitude oceans. A gap that is uniformly zero everywhere would mean the cycle carries no compressible structure worth modeling.
The strongest simple competitor is a phase-normalized stationary model. It subtracts the climatological mean and divides by the climatological standard deviation for each hour of day and day of year, then applies stationary theory or a standard compressor to the residual. It is strong because, unlike in communications, the cycle's phase in the atmosphere is known exactly from the sun. It will use the same climatology estimation period and the same data. Our bound beats it only if the cross-phase correlation structure, which normalization does not remove, yields a measurable rate reduction. Matched variance alone does not count.
Among existing compressors, the strongest are SZ3 in error-bounded mode and the coordinate-based neural compressor of Compressing multidimensional weather and climate data into neural networks. That paper reports usable compression ratios of 300× to 3,000× on ERA5. It reports a downstream WeatherBench forecast error increase of less than 2% when training on 790×-compressed data, and RMSE and MAE better than SZ3 above 400×. No reported result was found on Lacuna for SZ3's absolute rate on HRRR-resolution extremes. Both compressors will be run on identical blocks at matched mean squared error and at matched maximum error inside HR-Extreme event boxes. The neural compressor gets its published configuration, and SZ3 gets a sweep of error bounds. Here "beating" has a specific meaning: a compressor whose rate falls below our Gaussian upper bound is not a failure of the method, but a compressor that sits far above it reveals headroom.
Two ablations isolate the key ingredients. In the first, the cyclic bound is computed without bounded delay and phase alignment, by truncating the delay-free formula. In the second, the information-spectrum quantile is replaced by the mean rate when sizing the stream.
Why not just…?
Why not just phase-normalize the data and use ordinary stationary theory? This is the real threat. Because solar phase is known exactly, normalization might capture almost everything the almost-cyclostationary bound captures. The theory in The Distortion Rate Function of Cyclostationary Gaussian Processes says normalization removes variance modulation but not correlations between different phases, such as the link between morning moisture and afternoon convection. Whether those correlations carry bits in real data is an empirical question. The second baseline settles it directly. If the gap is negligible everywhere, we will report that the practical content of the theory reduces to this recipe plus a rigorous bound, and the cadence and spectrum results stand on their own.
Why not just histogram SZ3's per-block compressed sizes to size the stream? For sizing a buffer for one compressor on one archive, that empirical histogram is sufficient, and we will use it as the ground truth that our predicted quantile must match. What it cannot give is a compressor-independent target, the headroom between a compressor and the limit, or a prediction made before the data are generated. Such predictions would cover how the required rate shifts with output cadence, block phase or model resolution. If our predicted quantile does not track the empirical one across cadences, the spectrum component has no operational value, and we will say so.
Why not just use neural compression, which already reaches thousands-fold ratios? Its own authors report that it smooths extremes and takes about 8 hours to encode Compressing multidimensional weather and climate data into neural networks. That rules it out for bounded-delay streaming from runs like those in Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 GPUs with COSMO 5.0. Average ratio is also the wrong figure of merit when the failure blocks are the storms. Our audit reports its per-block error on HR-Extreme boxes, which tests this concern directly.
Why assume Gaussianity for precipitation? Because under squared error the Gaussian bound is an upper bound for any source with the same covariance, it remains a valid budget even when wrong. The cost is looseness, and the raw-versus-Gaussianized comparison measures how loose. If compressors on raw precipitation fall far below the bound, the looseness is large, and a non-Gaussian lower bound becomes the next problem to solve, which is the open problem the Gaussian papers leave.
Proposed Experiments
The first experiment is the cadence-sensitivity test, the atmospheric analogue of the 0.2%-shift result. On ERA5 hourly 2 m temperature, total column water vapor and precipitation, we compare blocks at 3-hour cadence with blocks at cadences perturbed by 0.2% to 2%. Both arms are produced by the same temporal interpolation so that interpolation error cancels. We predict a rate change far smaller than the 30% capacity swing in The Capacity of Memoryless Channels With Sampled Cyclostationary Gaussian Noise, because atmospheric memory smooths phase drift. A large swing would itself be a notable finding for archive design.
The second experiment is the block-phase test. On HRRR fields in HR-Extreme, we start blocks at different local solar times and measure both the bounded-delay rate and SZ3's actual rate. The bounded-delay theory predicts a best starting phase and a dependence on phase that shrinks as blocks lengthen.
The decisive experiment is the spectrum test. We compute the predicted $\epsilon$-quantile rate for $\epsilon$ between 0.01 and 0.1 and compare it with the empirical quantile of per-block compressed sizes from SZ3 at matched error. Agreement is measured as calibration error across cadences and phases. We then check whether the blocks in the upper tail of the predicted spectrum are enriched for HR-Extreme event boxes, using precision against the box labels.
The proposal succeeds if three things hold. The cyclic bound must beat phase normalization at convective sites. The predicted quantile must track the empirical one to within a few percent of rate. And tail blocks must be substantially enriched for labeled hazards. It fails informatively if phase normalization matches the bound, or if the predicted spectrum does not track any real compressor.
References
- The Rate-Distortion Function for Sampled Cyclostationary Gaussian Processes with Memory and with Bounded Processing Delay
- The Distortion Rate Function of Cyclostationary Gaussian Processes
- On the Rate-Distortion Function of Sampled Cyclostationary Gaussian Processes
- On the Rate-Distortion Function for Sampled Cyclostationary Gaussian Processes with Memory
- The Capacity of Memoryless Channels With Sampled Cyclostationary Gaussian Noise
- A formula for the capacity of the general Gel'fand-Pinsker channel
- A Lower and Upper Bound on the Epsilon-Uniform Common Randomness Capacity
- Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 GPUs with COSMO 5.0
- Compressing multidimensional weather and climate data into neural networks
- Diagnosing added value of convection-permitting regional models using precipitation event identification and tracking
- HR-Extreme: A High-Resolution Dataset for Extreme Weather Forecasting