When the Graph Is the Leak: Distance-Controlled Evaluation of Graph Precipitation Models on Sparse Gauge Networks

Rain gauges are scarce and clustered, so a growing class of models represents weather stations, satellite patches and meteorological variables as nodes in a graph. Each node's prediction is then informed by its neighbors, and some models even learn to add the connections they think are missing. These models are usually scored on a later time period at the same fixed locations they were trained on. In that setting, a graph gives the model a shortcut: it can copy what nearby gauges report at the same moment, or recall what a particular station usually does, without learning anything that carries over to the ungauged valleys and ridges where estimates are actually needed. Machine learning evaluation has a name for scores inflated by exposure to test material: benchmark contamination. This proposal argues that, for graph-based precipitation models, the graph itself acts as a contamination channel. It sets out to measure how large that effect is.

Concretely, we build an evaluation that regenerates its train/test splits on demand. It holds out gauge locations at controlled distances from the nearest training gauge, removes or rewires graph edges under controlled interventions, and reports skill as a curve over distance instead of a single number. The claim, stated plainly: the reported skill of graph precipitation models can be split into three parts. One is station memorization, one is copying from neighbors through edges, and one is transferable skill. Only the third part should count, and current protocols cannot tell these parts apart.

Core Claim and What Is New

Three bodies of work motivate the claim but do not address it. DAHG builds dynamic heterogeneous graphs over stations, MODIS vegetation-index patches and weather variables. It uses a reinforcement-learning agent to add missing edges and reports MAE 1.28 mm/h and R² 0.881 on ERA5-Land. Its split is purely chronological (2010–2017 train, 2020 test) over fixed grid locations, and the target is ERA5-Land. ERA5-Land is a smoothed reanalysis (a physics-model reconstruction of past weather), not gauge truth. That combination is close to ideal for neighbor copying.

Consistent Validation for Predictive Methods in Spatial Settings shows that holdout validation on clustered, fixed sensor sites gives biased error estimates. It proposes Spatial Nearest Neighbors (SNN) reweighting, which is consistent as validation density increases. The authors explicitly leave extrapolation into regions without sensors open, and they do not study graph models.

Work on knowledge-graph dependency shows that downstream models treat edges as ground truth, so noisy or inferred edges cascade into confident errors (Shen et al.; Xian et al.). No one has asked what this means when the edges are learned from, and scored against, the same spatial field.

From the contamination literature we borrow a method rather than a domain. DyVal generates reasoning tasks from graphs at controlled depth, so that a memorized static test set cannot hide a collapse in skill. For example, it reports GPT-4 losing 13.6% on reachability tasks when premises are given in reversed order. KBE-DME similarly re-selects knowledge subgraphs to regenerate visual question answering items at controlled hop counts. We carry this idea over, with "hops" replaced by physical distance to the nearest training gauge. Unlike both papers, we need no generative model to synthesize items, because dense radar-gauge fields supply ground truth at every location we choose to hold out. Unlike DAHG, we treat the graph's edges as the variable under test instead of as trusted structure.

Research Program

The central object is a split generator. It is built on a dense truth field in which a sparse "pseudo-gauge" network is sampled with realistic clustering, for example denser near low-elevation population centers. Each draw holds out target locations stratified by $d/\ell$. Here $d$ is the distance from a held-out location to its nearest training gauge, and $\ell$ is the empirical decorrelation length of hourly precipitation, estimated from the variogram of the training period. Because splits are resampled per evaluation run, no fixed spatial test set can be memorized. This is the same guarantee DyVal gives for reasoning items.

For any model we report a skill-versus-distance curve. We also report a leakage gap, defined as

$$\Delta = S_{\text{chron}} - S_{\text{held}}(d/\ell > 1).$$

In this definition, $S_{\text{chron}}$ is skill (for example, one minus the normalized MAE) under the usual chronological split at training locations. $S_{\text{held}}(d/\ell > 1)$ is skill at held-out locations lying beyond one decorrelation length.

We then apply three interventions, each targeting one channel of shortcut learning. The first, identity shuffling, permutes learned node embeddings or station identifiers among stations with similar elevation. It measures memorization of individual stations. The second, degree-preserving edge rewiring at controlled rates, keeps each node's number of edges fixed while changing which nodes are connected. It measures dependence on specific edges. The third, same-timestep neighbor masking, hides co-temporal neighbor observations while keeping lagged ones. It measures copying of coincident rainfall.

For DAHG-style models we also track the edges the reinforcement-learning agent adds. Specifically, we record whether they preferentially link held-out nodes to the training nodes most correlated with them. That behavior would be the graph analogue of test-set leakage.

The primary testbed is RainNet. RainNet provides 17 years of hourly StageIV radar-gauge fields at 4 km over the Eastern United States, dense enough to hold out any location. For terrain transfer we use APHRODITE gauge-based precipitation over Asia, as used in the GA-GWR terrain study. There we sub-sample to Nepal- and Tianshan-like densities, the sparse mountain settings documented in Nepal IMERG evaluation and Tianshan machine-learning precipitation estimation. Finally, we re-run on DAHG's ERA5-Land setup to check whether its reported numbers survive the new protocol.

Baselines

The naive reference point is climatology plus inverse-distance weighting from training gauges. It has no learning and no graph. Any model that fails to beat it beyond $d/\ell = 1$ has no demonstrated transferable skill.

The strongest classical competitor is ordinary kriging with an elevation trend. RainNet found that kriging outperformed SRGAN even though it blurs sharp structures. Kriging is also the method whose skill should decay with distance in a principled way, which makes it the natural comparison for our curves. We fit it on exactly the same pseudo-gauges for each split, with the variogram estimated only from training data.

The non-linear competitor is Random-Forest residual correction on elevation, slope and land-surface predictors, as in Tianshan machine-learning estimation. We also include Kalman-filter fusion of satellite fields with gauges, following the Jinghe Basin TRMM assimilation.

The strongest existing graph method is DAHG. Its reported MAE of 1.28 mm/h and R² of 0.881 on ERA5-Land under chronological splits are the numbers we first reproduce under the original protocol, to within run-to-run variance, before applying ours. For RainNet-derived fields, RainNet reports RMSE 0.312 for its best implicit sequence model. That is a downscaling result rather than a sparse-gauge result, so we use it only to confirm our data pipeline.

All learned models receive the same inputs, the same split draws (at least 20 per distance band), and the same hyperparameter budget, tuned on a validation fold that is itself distance-controlled.

Two ablations isolate the key ingredient. The first is DAHG with its edge-completion agent disabled, using a fixed k-nearest-neighbor graph. The second is DAHG with the fixed graph and no node identity embeddings. For a graph model to "beat" kriging in the sense that matters, its held-out skill must be significantly higher for $d/\ell > 1$ across split draws, not merely at training locations.

Why not just…?

Why not just use spatial block cross-validation or SNN reweighting? This is the most serious alternative, and in part we adopt it: a single block split is one point on our distance curve. It falls short in three ways. A fixed block, once published and reused, becomes a static test set again, which is the failure DyVal documents for reasoning benchmarks. Blocks also mix distances, so a good block score can hide sharp decay at the block's interior. And SNN, by its authors' own statement, guarantees consistency under infill, not extrapolation. We compare the block-CV estimate and the SNN-reweighted estimate against the dense-truth error at held-out locations. If block CV already predicts that error as accurately as the full distance curve, the curve adds little beyond diagnosis. In that case the contribution narrows to the intervention analysis, and we will say so.

Why not just drop the graph and use kriging or Random Forest? That may well be the right answer, and the experiments are designed to show it if so. If graph models match kriging beyond one decorrelation length, and their advantage lives entirely at training locations and disappears under neighbor masking, the finding is that the graph's measured benefit is leakage. That is a publishable negative result that directly qualifies DAHG's reported gains.

Why not just remove station identity embeddings and call it fixed? Identity is only one of three channels. Co-temporal neighbor copying and inferred edges that attach test nodes to their most correlated training nodes both survive without embeddings. The two-stage ablation above measures how much of the leakage gap each channel accounts for.

Proposed Experiments and Decisive Checks

The first check is whether $\Delta$ is large for graph models and small for kriging on the RainNet-derived fields. We expect the gap to be largest for DAHG with edge completion, because the knowledge-graph sensitivity results predict that inferred edges will be trusted uncritically.

The second check is whether degree-preserving rewiring of 20–50% of edges degrades graph-model skill at training locations far more than at distant held-out locations. That pattern would indicate that edges carry location-specific shortcuts rather than transferable physical relations.

The third check is whether edges added by the reinforcement-learning agent concentrate on pairs with high same-timestep correlation, measured against a random-edge null.

The fourth check is whether these conclusions transfer to the APHRODITE mountain subsets. There, sparse and clustered gauges should amplify the gap. If the gap does not grow with sparsity, our explanation is wrong.

A clean falsification of the whole thesis would be graph models keeping a significant advantage over kriging beyond $d/\ell = 1$ under neighbor masking. That outcome would show real transferable relational skill, and the protocol would then serve as the evidence for it.