Distribution-Free Uncertainty Quantification for Weather Forecasting: Conformal Prediction Under Spatiotemporal Dependence

MariyaarXiv 2026 · arXiv

TL;DR

Conformal prediction offers a robust way to attach uncertainty intervals to weather forecasts without assuming a specific error distribution. However, this paper reveals that while standard methods achieve the target "average" coverage, they suffer from massive daily fluctuations and clustered failures due to the inherent spatiotemporal correlations in weather. The author demonstrates that "systemic" day-level error scales drive most of this unreliability, pointing toward a need for new, dependence-aware calibration methods.

What Problem Does This Paper Address?

As machine learning models become competitive with traditional numerical weather prediction, we need reliable ways to quantify their uncertainty. Conformal Prediction (CP) has emerged as a favorite because it provides a "distribution-free" guarantee: a 90% prediction interval will contain the true value 90% of the time, regardless of the model's underlying distribution.

The catch? This guarantee is marginal. It tells you about the long-run average at a single location. It doesn't tell you how the model behaves across an entire grid on a specific Tuesday. In weather, errors are rarely independent; if a forecast misses in one city, it likely misses in the next city over (spatial dependence), and it might miss again tomorrow (temporal dependence). This leads to "systemic failures" where a 90% interval might cover only 70% of the globe on one day and 99% on another. This paper quantifies this gap between theoretical marginal coverage and practical daily reliability.

Methods

The author benchmarks eight different interval-construction procedures:

  • Standard Baselines: Gaussian (RMSE-based) and Split Conformal (pooled and per-cell).
  • Context-Aware Methods: Per-cell + Season (Mondrian) calibration.
  • Online Methods: Adaptive Conformal Inference (ACI) and Rolling Windows (90 and 365 days), which update intervals based on recent errors.

To measure the impact of dependence, the paper introduces several key diagnostics:

  • Spatial Overdispersion Index (VIF): A "Variance Inflation Factor" that compares the actual daily variance of grid-wide coverage to what you'd expect if cells were independent.
  • Miss Persistence: The probability of a "miss" today given a "miss" yesterday, compared to a randomized null.
  • Oracle Day-Scale: A theoretical test that normalizes errors by a daily global factor to see if "knowing how hard the day is" fixes the overdispersion.

Datasets

The study uses the ERA5 reanalysis dataset for 2m temperature (T2m) on a 5.625° equiangular grid. It evaluates two different forecasters:

  1. A custom, "deliberately modest" Convolutional Neural Network (CNN).
  2. The European Centre for Medium-Range Weather Forecasts (ECMWF) High-Resolution (HRES) operational forecast.

The data is split into training (1979–2015), calibration (2017–2018), and testing (2019–2021) periods.

Key Insights and Ideas

  • Marginal coverage is a low bar: Almost all per-cell methods successfully hit the 90% coverage target on average. However, the VIF shows that the daily fraction of cells covered is 3 to 10 times more variable than it should be.
  • The "Day-Scale" is the culprit: The "Oracle" experiment is a major "aha" moment. By simply dividing each day's errors by that day's mean error, the author removed 81–98% of the excess spatial dispersion. This proves that the unreliability isn't just about local correlations; it's driven by broad, grid-level fluctuations in forecast difficulty.
  • Skill doesn't solve the problem: Even the highly sophisticated HRES model—which is much more accurate than the simple CNN—exhibits similar spatial overdispersion and temporal clustering. This suggests that dependence is a fundamental property of the atmosphere, not just a symptom of a weak model.
  • The Lead-Time Trap: As lead times increase (e.g., from 6 hours to 10 days), the forecaster's skill drops toward climatology, and the clustering of misses becomes significantly more pronounced.

Key Results

The quantitative results highlight the trade-offs between different methods. While per-cell methods are generally better than pooled ones, they still struggle with dependence.

Table 1. Area-weighted RMSE (K) of 2 m temperature over the test period.

As seen in Table 1 above, the CNN forecaster beats persistence and climatology up to about 3–5 days, after which its skill degrades.

Table 2. Coverage, interval score and VIFhom at α = 0.1, with 95% block-bootstrap intervals over test days (calibration set held fixed). Nominal coverage is 0.90.

Table 2 shows that while most methods achieve near 0.90 coverage, the VIF (overdispersion) is high across the board. For example, at a 120h lead, the VIF for per-cell split conformal is ~6.04, meaning the daily coverage variance is six times higher than the independent-cell ideal.

Limitations and Future Work

The author acknowledges several limitations:

  • Scope: The study focuses on one variable (T2m) at a relatively coarse resolution. Results might differ for precipitation or finer grids.
  • Target: Using reanalysis (ERA5) as a target is a proxy for reality; results might vary when compared against actual station observations.
  • Predictability: While the "Oracle" shows that a day-scale factor exists, predicting it at the time of issuance proved difficult, yielding only a ~0.6% improvement in interval scores.

Future Directions: The paper concludes that we need dependence-aware methods. Instead of calibrating each cell in isolation, future work should treat the entire forecast field as the unit of calibration. Proposed ideas include using ensemble spread to estimate the "day-scale" error and developing spatially coupled versions of Adaptive Conformal Inference.

Summary updated

Abstract

Conformal prediction gives finite-sample, distribution-free marginal coverage and has begun to be applied to gridded weather forecasts. Marginal coverage concerns the long-run fraction of covered points at a location, not how coverage behaves jointly across space and time on a given day. I benchmark eight interval-construction procedures (Gaussian, pooled and per-cell split conformal, per-season, adaptive conformal inference and rolling windows) for gridded 2 m temperature forecasts at eight lead times from 6 hours to 10 days, using ERA5 at 5.625 degree resolution, a deliberately modest convolutional forecaster and ECMWF's operational HRES, with calibration in 2017-2018 and testing in 2019-2021. Alongside coverage, width and interval score, I report dependence diagnostics: a spatial overdispersion index, quantiles of daily spatial coverage and miss persistence relative to a shuffled-days null. Per-cell methods reach near-nominal marginal coverage, yet the day-to-day variance of the fraction of cells covered is roughly 3 to 10 times what independent cells would give, and misses cluster in time; both effects grow with lead time for the CNN, also appear for HRES, and are shared by all methods. An oracle that divides each day's normalised errors by that day's mean removes 81 to 92 percent of the excess dispersion for the CNN and 93 to 98 percent for HRES, so days of large errors across much of the grid account for most of it. A day-scale predicted from seasonal terms and already-verified errors lowers the overdispersion index without widening intervals but improves interval score by at most 0.6 percent; on HRES from 12 hours to 3 days it has the lowest overdispersion index while ACI or season-conditional calibration has the best interval score, so no method is best on both. The study covers one variable, one coarse grid, a reanalysis target and two forecasters.
BibTeX
@misc{mariya2026distribution,
  author = {Mariya},
  title = {{Distribution-Free Uncertainty Quantification for Weather Forecasting: Conformal Prediction Under Spatiotemporal Dependence}},
  year = {2026},
  eprint = {2611.24329},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24329},
}
Figure