TL;DR
Conformal prediction offers a robust way to attach uncertainty intervals to weather forecasts without assuming a specific error distribution. However, this paper reveals that while standard methods achieve the target "average" coverage, they suffer from massive daily fluctuations and clustered failures due to the inherent spatiotemporal correlations in weather. The author demonstrates that "systemic" day-level error scales drive most of this unreliability, pointing toward a need for new, dependence-aware calibration methods.
What Problem Does This Paper Address?
As machine learning models become competitive with traditional numerical weather prediction, we need reliable ways to quantify their uncertainty. Conformal Prediction (CP) has emerged as a favorite because it provides a "distribution-free" guarantee: a 90% prediction interval will contain the true value 90% of the time, regardless of the model's underlying distribution.
The catch? This guarantee is marginal. It tells you about the long-run average at a single location. It doesn't tell you how the model behaves across an entire grid on a specific Tuesday. In weather, errors are rarely independent; if a forecast misses in one city, it likely misses in the next city over (spatial dependence), and it might miss again tomorrow (temporal dependence). This leads to "systemic failures" where a 90% interval might cover only 70% of the globe on one day and 99% on another. This paper quantifies this gap between theoretical marginal coverage and practical daily reliability.
Methods
The author benchmarks eight different interval-construction procedures:
- Standard Baselines: Gaussian (RMSE-based) and Split Conformal (pooled and per-cell).
- Context-Aware Methods: Per-cell + Season (Mondrian) calibration.
- Online Methods: Adaptive Conformal Inference (ACI) and Rolling Windows (90 and 365 days), which update intervals based on recent errors.
To measure the impact of dependence, the paper introduces several key diagnostics:
- Spatial Overdispersion Index (VIF): A "Variance Inflation Factor" that compares the actual daily variance of grid-wide coverage to what you'd expect if cells were independent.
- Miss Persistence: The probability of a "miss" today given a "miss" yesterday, compared to a randomized null.
- Oracle Day-Scale: A theoretical test that normalizes errors by a daily global factor to see if "knowing how hard the day is" fixes the overdispersion.
Datasets
The study uses the ERA5 reanalysis dataset for 2m temperature (T2m) on a 5.625° equiangular grid. It evaluates two different forecasters:
- A custom, "deliberately modest" Convolutional Neural Network (CNN).
- The European Centre for Medium-Range Weather Forecasts (ECMWF) High-Resolution (HRES) operational forecast.
The data is split into training (1979–2015), calibration (2017–2018), and testing (2019–2021) periods.
Key Insights and Ideas
- Marginal coverage is a low bar: Almost all per-cell methods successfully hit the 90% coverage target on average. However, the VIF shows that the daily fraction of cells covered is 3 to 10 times more variable than it should be.
- The "Day-Scale" is the culprit: The "Oracle" experiment is a major "aha" moment. By simply dividing each day's errors by that day's mean error, the author removed 81–98% of the excess spatial dispersion. This proves that the unreliability isn't just about local correlations; it's driven by broad, grid-level fluctuations in forecast difficulty.
- Skill doesn't solve the problem: Even the highly sophisticated HRES model—which is much more accurate than the simple CNN—exhibits similar spatial overdispersion and temporal clustering. This suggests that dependence is a fundamental property of the atmosphere, not just a symptom of a weak model.
- The Lead-Time Trap: As lead times increase (e.g., from 6 hours to 10 days), the forecaster's skill drops toward climatology, and the clustering of misses becomes significantly more pronounced.
Key Results
The quantitative results highlight the trade-offs between different methods. While per-cell methods are generally better than pooled ones, they still struggle with dependence.
As seen in Table 1 above, the CNN forecaster beats persistence and climatology up to about 3–5 days, after which its skill degrades.
Table 2 shows that while most methods achieve near 0.90 coverage, the VIF (overdispersion) is high across the board. For example, at a 120h lead, the VIF for per-cell split conformal is ~6.04, meaning the daily coverage variance is six times higher than the independent-cell ideal.
Limitations and Future Work
The author acknowledges several limitations:
- Scope: The study focuses on one variable (T2m) at a relatively coarse resolution. Results might differ for precipitation or finer grids.
- Target: Using reanalysis (ERA5) as a target is a proxy for reality; results might vary when compared against actual station observations.
- Predictability: While the "Oracle" shows that a day-scale factor exists, predicting it at the time of issuance proved difficult, yielding only a ~0.6% improvement in interval scores.
Future Directions: The paper concludes that we need dependence-aware methods. Instead of calibrating each cell in isolation, future work should treat the entire forecast field as the unit of calibration. Proposed ideas include using ensemble spread to estimate the "day-scale" error and developing spatially coupled versions of Adaptive Conformal Inference.
Summary updated