Machine-learning downscaling models turn coarse weather or climate fields, with grid cells of 25 to 100 km, into kilometre-scale maps. They now report near-perfect scores. One 10-billion-parameter vision transformer reports coefficients of determination ($R^2$) of 0.98 for precipitation and 0.99 for temperature on 7 km benchmarks (ORBIT-2). These evaluations share a quiet flaw that parallels benchmark contamination in language models, where test questions leak into training data. Every downscaling benchmark trains and tests on the same fixed patch of Earth. The mountains, coastlines, lakes and cities in the test years are exactly those of the training years, and so is each location's typical seasonal cycle. A network can therefore score very well by memorizing a high-resolution "map" of each place's climate and making small adjustments. It does not need to translate the day's large-scale weather into local detail. For temperature, most of the variance a model has to explain is spatial and seasonal, so a headline $R^2$ of 0.99 says little about whether the model actually used the coarse input.
We propose to measure this directly. We split downscaling skill into a static part, which any predictor that memorizes location and season could achieve, and a dynamic part, which can only come from information in the coarse atmospheric input. We then track the dynamic part as test days move away from anything seen in training. The work transfers three ideas from contamination research. The first is a memorization-only reference, which plays the role of an "open book" baseline. The second is input-substitution sensitivity: feed the model the wrong day's input, in the spirit of the perturbation tests in PEARL. The third is performance as a function of distance to the nearest training exemplar, the regression analogue of the rule-versus-exemplar question asked in Do Tabular Foundation Models Learn Rules or Memorize Exemplars?. The deliverable is a scoring protocol and a public audit of current generative downscalers.
Core Claim
Our hypothesis is that a large fraction of the reported skill of learned downscalers on fixed-domain benchmarks is reproducible by a predictor that sees nothing of the day's fine-scale information. That predictor is the training-period high-resolution climatology for each location and day of year, plus the interpolated coarse anomaly. We further expect the remaining dynamic skill to be concentrated on days whose coarse state has a close historical analog, and to decay sharply on novel days and extremes. This is the downscaling version of the "collapse off-distribution" pattern documented for language models by DyVal and Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?.
The claim is falsifiable in both directions. If dynamic skill turns out large and roughly flat across analog distance, the memorization concern is unfounded, and the field gains a stronger evidential basis for its numbers.
Relation to Prior Work
The downscaling papers motivate the gap, because each holds out time or model realization but never the static map. Universal Diffusion-Based Probabilistic Downscaling trains a conditional diffusion model, a generator that produces samples by repeatedly denoising random noise, on paired ERA5 (about 25 km) and CERRA (about 5 km) reanalysis for 2014–2023. It then evaluates on station observations from July 2024 to June 2025, which is a careful temporal holdout over the same European stations and terrain. Climate Downscaling with Stochastic Interpolants learns a direct transport from EC-Earth (about 110 km) to HCLIM (about 12 km). It tests on later years and on an unseen ensemble member, but over the identical EURO-CORDEX domain. A Generative Framework for Probabilistic, Spatiotemporally Coherent Downscaling conditions a score-based prior on COSMO-REA6 at 6 km, and ORBIT-2 scales transformers across PRISM, DAYMET and IMERG. None of these works separates location-and-season memory from input-driven skill. We do not propose a new downscaler. We re-score these model families under a protocol that makes that separation.
WeatherBench constrains the design. It shows that error metrics reward blurred averages, so our dynamic-skill measure must be probabilistic and spectrum-aware, not error-only. ClimaX argues that randomized pretraining objectives discourage memorizing fixed patterns, but it offers no test of that argument; our protocol would supply one.
The contamination papers supply the method. PEARL's perturbation-sensitivity idea rests on a simple observation: memorized items break under small input changes, while learned rules degrade smoothly. We turn it into coarse-input substitution. The tabular exemplar-versus-rule study motivates our analog-distance curve. Classical constructed-analog downscaling, which in the file's account is Gershunov and colleagues' method of combining historical fine-scale fields whose coarse states match today's, gives us an explicit pure-exemplar predictor to compare against. P⁴Bench addresses leakage through cryptographic withholding of answer keys. That approach does not apply here, because the leak is the static geography itself and not a scraped answer key.
Research Program
The central quantity is a dynamic skill score measured against the strongest memorization-only reference:
$$ D = 1 - \frac{\mathrm{CRPS}_{\text{model}}}{\mathrm{CRPS}_{\text{mem}}}, $$
Here CRPS is the continuous ranked probability score. It measures the distance between a predicted distribution and the observed value, and reduces to absolute error for a deterministic forecast. $\mathrm{CRPS}_{\text{mem}}$ is the score of the memorization-only reference. That reference is the training-period high-resolution climatology at each pixel and day of year, plus the bicubically interpolated coarse anomaly. Its spread is set to the training-period residual distribution at that pixel and season. By construction, it uses the coarse input only through smooth interpolation, so any skill beyond it must come from the model turning large-scale weather into local structure. We also report $D$ computed with root mean squared error, and the ratio of fine-scale spectral power between the model's output and the truth, so that sharpness cannot hide behind the metric.
The program has three components. The first is the anomaly audit: we compute $D$ for each model family, variable and season, and in particular we recompute the headline-style $R^2$ against the memorization reference. The second is an input-substitution test. We give the model the coarse field from a randomly chosen other date in the same season, with static inputs unchanged, and measure how much skill against the true date's target survives. A model whose outputs barely change is running on the map. We also apply graded versions that blend the true and swapped inputs, which traces a sensitivity curve in the manner of PEARL. The third is an analog-distance curve. For each test day we compute the distance from its coarse state to the nearest training-period coarse state, in a principal-component space of the large-scale fields. We then plot $D$ against that distance, with separate curves for extremes above the 99th percentile. We summarize each curve by the distance at which $D$ falls to zero. Beyond that point the model is no better than memorization.
To obtain test cases that cannot possibly have been memorized, with ground truth available, we add a procedural track on the 2D Navier-Stokes setting used by Fourier Neural Operators for Arbitrary Resolution Climate Data Downscaling. We extend it with a static forcing field that stands in for terrain. We train on a small fixed set of forcing fields and test on freshly sampled ones, in the style of DyVal. This lets us check that $D$ and the analog-distance curve correctly rank a model trained to memorize against one that generalizes, in a setting where the answer is known.
Baselines
The naive reference is bicubic interpolation of the coarse field, with a quantile-mapping bias correction as used in A Generative Framework for Probabilistic, Spatiotemporally Coherent Downscaling. It anchors the bottom of every table. On the Navier-Stokes track, DFNO reports 2x-downscaling mean squared error of 0.0015 against 0.0042 for a convolutional network and 0.0044 for a generative adversarial network. No reported result on Lacuna was found for bicubic interpolation alone on that data. We will reproduce DFNO's reported figures before running our tests, to confirm that the setup matches.
The memorization-only reference is the key ablation, and its role differs from the others. It is the predictor that isolates what the map alone achieves, and $D$ is defined relative to it. We do not expect a learned model to lose to it. What matters is by how much the model beats it, and where it stops beating it. No reported result on Lacuna was found for this reference on ERA5→CERRA, and producing that number is itself a contribution.
The pure-exemplar baseline is constructed-analog downscaling. It retrieves the k training days with the closest coarse states and combines their fine-scale fields, with k tuned on the 2023 validation year. If a deep model's analog-distance curve lies on top of this baseline's curve, the deep model is behaving as a retrieval system.
The strongest learned competitor is the conditional diffusion recipe of Universal Diffusion-Based Probabilistic Downscaling. That paper reports CRPS skill improvements of 15–30% over upstream forecasts in the short range. It is the strongest competitor here because it uses the cleanest temporal holdout in the file and is cheap to retrain, at about 8 hours on 8 GPUs by its own report. We also run a stochastic-interpolant model following CDSI, for which no numerical score was found on Lacuna, and a deterministic U-Net. We cite the ORBIT-2 figures of $R^2$ = 0.98 for precipitation and 0.99 for temperature as the headline numbers our anomaly audit reinterprets. We cannot retrain a 10-billion-parameter model, so ORBIT-2 enters only through its published numbers and any public checkpoint.
All learned models are trained on the same ERA5→CERRA pairs for 2014–2022, validated on 2023, and tested on 2024–2025 gridded data and station observations. Each receives the same GPU budget and the same hyperparameter search size. A second ablation retrains each model without static inputs (orography, land-sea mask, coordinates) and on random crops with shuffled positions. This measures how much of the model's static-map advantage it rebuilds internally when it is not handed the map.
"Beating" in this study means something specific. A model demonstrates dynamic skill if $D$ is significantly above zero under a block bootstrap by week, and if $D$ stays positive beyond the analog distance at which the constructed-analog baseline's advantage over the memorization reference reaches zero.
Why not just…?
Why not just use a strict temporal holdout? Universal Diffusion already does this, and it rules out the model having seen the test days themselves. It cannot remove the static map, which is identical in 2024 and 2014. Our input-substitution test directly measures how much post-cutoff skill survives when the day's coarse information is removed. That surviving fraction is exactly what a temporal holdout cannot see.
Why not just test on an unseen region? Transfer across continents mixes memorization with a genuine change of climate regime. The Universal Diffusion paper itself lists transfer beyond Europe as unresolved, and ClimaX needed regional fine-tuning. A failure on a new continent would therefore not tell us why the model failed. Our tests keep the region and the climate regime fixed and vary only whether the day's input information is available. The procedural track supplies novel "terrain" where the physics is held constant.
Why not just score anomalies against climatology? This is the serious threat, because it is most of our method. We adopt it rather than argue with it. Two additions go beyond it. The reference includes the interpolated coarse anomaly, so it is the strongest predictor available without learned fine-scale structure, not plain climatology. And the substitution and analog-distance tests explain where dynamic skill comes from, rather than giving only its total. If $D$ alone ranks the models identically to the full protocol, we will report the simpler score as sufficient.
Why not just use classical downscaling, which cannot be contaminated? Classical methods are not immune to this particular problem, because constructed analogs are memorization by design. The file's evidence, including WeatherBench's linear-regression error and the blurring noted by CDSI, shows they are also less accurate. We include them as calibration points for what pure retrieval looks like, not as the answer.
Decisive Checks
The first result to look at is the input-substitution retention for temperature. It is the fraction of each model's skill above bicubic interpolation that survives when the model is fed a wrong same-season date. High retention for the diffusion model would confirm the core claim. Near-zero retention would refute it for that model.
The second is the analog-distance curve. We will report the distance at which each model's $D$ reaches zero, and whether the learned models' curves are distinguishable from the constructed-analog baseline, both overall and for the most extreme 1% of precipitation days.
The third is the procedural validation. On Navier-Stokes with fresh forcing fields, a model trained deliberately on two forcing fields must show a sharp collapse in $D$, and a model trained on hundreds must degrade smoothly. If the protocol cannot separate these two known cases, we will revise it before interpreting the real-data audit.
Station-level results on SYNOP and METAR provide an independent check. Those sites add microclimates that CERRA itself does not resolve, so a model that has merely memorized CERRA's map should gain no advantage there.
References
- ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling
- Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis
- Do Tabular Foundation Models Learn Rules or Memorize Exemplars?
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Universal Diffusion-Based Probabilistic Downscaling
- Climate Downscaling with Stochastic Interpolants (CDSI)
- A Generative Framework for Probabilistic, Spatiotemporally Coherent Downscaling of Climate Simulation
- WeatherBench: A Benchmark Data Set for Data-Driven Weather Forecasting
- ClimaX: A foundation model for weather and climate
- $\text{P}^4$Bench: Contamination-Proof, Publicly Verifiable, and Privacy-Preserving LLM Evaluation via Zero-Knowledge Proofs
- Fourier Neural Operators for Arbitrary Resolution Climate Data Downscaling