When medical machine learning splits data by record instead of by patient, models learn to recognize each patient rather than the disease, and the reported accuracy is an illusion. Kilometer-scale weather data has the same flaw, and it is largely unexamined. A single thunderstorm covers hundreds of 1–3 km grid cells and lasts for hours. Each storm has its own moisture, its own rain intensity, and its own position relative to terrain. When a model is trained on individual grid cells, pixels from one storm end up in both the training and test sets. The model then scores well by recognizing the storm, much as a model scores well by recognizing a patient. We propose to treat the tracked storm as the unit of statistical independence, the analogue of the patient, and to apply this in two places. The first is evaluation: cross-validation folds are built so that each storm falls entirely on one side. The second is training: a random forest (an average of many decision trees, each fit on a resampled copy of the data) resamples whole storms rather than pixels.
The training change is the part that has not been done. A standard random forest draws a bootstrap sample of pixels. Every tree therefore sees about 63% of the pixels of nearly every storm. Two consequences follow. The trees are highly correlated, because they all memorize the same storms. The "out-of-bag" error, the forest's built-in held-out estimate, is also leaky, because each out-of-bag pixel has storm-mates inside that tree's training sample. Resampling whole storms fixes both problems at once. Our central claim is testable: on convective precipitation downscaling, storm-grouped bagging produces an out-of-bag error close to the error on a truly held-out future period, and it gives equal or lower deployment error on heavy-rain events than pixel bagging, at the same tree count and compute.
Why this matters and what prior work leaves open
Three groups of papers motivate the gap. The medical leakage literature shows how large the illusion can be. In cuffless blood-pressure estimation, Examining the challenges of blood pressure estimation via photoplethysmogram found that the error standard deviation rose from 8.01 to 23.1 mmHg once subject overlap was removed. Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL found the same inflation for record-level ECG splits. Both papers fix only the evaluation split; neither changes how the model is trained.
The geospatial literature brings the problem to tree ensembles. Cross-validation for Geospatial Data and Model Evaluation for Geospatial Problems showed that a random forest evaluated with ordinary cross-validation was 43% overly optimistic on California Housing. Their buffered, importance-weighted method (IBCV) excludes training points near the test fold and reweights the rest to correct the resulting distribution shift. It brought the estimate to within 9% of the true error. We reuse IBCV as a baseline. It differs from our approach because it defines neighbors by static 2D distance, while storms move and change over their lifetimes.
The atmospheric literature supplies the grouping unit and the application. Diagnosing added value of convection-permitting regional models built an object-based tracker that follows rain events above 0.2 mm per 6 h through space and time. It used the tracker only to diagnose physical models, showing for example that 4 km WRF cut storm-size bias to 33% compared with 150–220% at 12 km. We repurpose the same tracker to define groups for machine learning. HR-Extreme clusters severe-weather reports into event boxes on 3 km HRRR data, but it does not split its train and test sets by event. It also evaluates only deep networks and numerical weather prediction, with no tree baselines.
One paper is a direct source of concern. Snow Depth Fusion Based on Machine Learning Methods for the Northern Hemisphere reports a random forest reaching R² = 0.91 from randomly sampled station data. We read that number as a likely leakage-inflated figure rather than as evidence against our concern. Its documented underestimation of snow deeper than 200 cm is the tail failure we expect to get worse under honest splits.
Core Claim
The ensemble-variance identity explains why storm-grouped bagging should help. Suppose $B$ trees each have prediction variance $\sigma^2$ and pairwise correlation $\rho$. The variance of their average is then
$$\mathrm{Var}(\bar T) = \rho\,\sigma^2 + \frac{1-\rho}{B}\,\sigma^2.$$
When pixels within a storm are nearly duplicates, the effective sample size is the number of storms $S$, not the number of pixels. Pixel bootstrapping gives every tree almost all $S$ storms, which pushes $\rho$ toward 1 and leaves a high variance floor that adding trees cannot remove. Storm bootstrapping gives each tree about 63% of the storms, which restores the diversity that bagging was designed to create.
We pair this with a second ingredient: a minimum-storm-diversity rule for leaves. A leaf may not split unless each child contains pixels from at least $k$ distinct storms. This stops individual trees from carving out leaves that describe a single storm. Out-of-bag error is computed per storm, so it becomes a leakage-free estimate at no extra cost.
Research Program
The main task is per-pixel daily precipitation downscaling over the contiguous United States, following the setup in Downscaling Extreme Precipitation With Wasserstein Regularized Diffusion. Inputs are ERA5 (31 km) and CPC (55 km) fields interpolated to each target pixel, together with terrain height and its local gradients. The target is MRMS 1 km precipitation.
Storm objects are built by running the tracker from Diagnosing added value of convection-permitting regional models on MRMS. Dry pixels are assigned to the nearest storm in space and time, or to calendar-day "background" groups.
A second task tests generality. We use HR-Extreme's 1-hour prediction of surface variables, with its DBSCAN event boxes serving as the groups, and train gradient-boosted and random-forest regressors per pixel on HRRR predictors.
The anchor for every comparison is a deployment error that no cross-validation scheme can see. Models are fit on the training period and region, then scored on a later, disjoint period. Each cross-validation estimate is judged by its relative bias against this deployment error, which is the same criterion the geospatial papers use. Each training method is judged by its deployment error overall and on events above the 99th percentile of rain, using root-mean-square error, critical success index (the fraction of correctly forecast heavy-rain pixels among all pixels where heavy rain was forecast or observed), and object-level storm size and intensity bias.
Baselines
The naive reference point is a standard random forest with pixel bootstrapping, evaluated by random K-fold cross-validation and by its own out-of-bag error. This is how the snow-depth fusion study was set up, and we expect it to be strongly optimistic. On the geospatial side, the analogous optimism measured on Lacuna is 43%, reported in Cross-validation for Geospatial Data.
The strongest existing evaluation method is IBCV from the same paper. We run it on identical forests. Its buffer radius is chosen from the empirical autocorrelation range of the residuals, and density ratios are estimated with RuLSIF as the original authors did. The best reported result is an estimate within 9% of the true holdout error on California Housing. Our storm-isolated cross-validation beats it if its relative bias against the deployment error is smaller on both tasks, with confidence intervals obtained by resampling storms.
The strongest simple training-side competitor is a pixel-bootstrap forest with stronger regularization: a minimum leaf size and a maximum depth tuned under storm-isolated validation. This tests whether grouping matters or only shrinkage does.
The strongest model-class competitor on extremes is a deep network. On HR-Extreme, HR-Heim reports RMSE 1.60, against 3.27 for numerical weather prediction and 10.42 for Pangu. For WassDiff, no reported numerical result was found on Lacuna, so we retrain a convolutional downscaler on our splits.
Ablations isolate each ingredient. We test storm bootstrapping without the leaf-diversity rule, the leaf rule without storm bootstrapping, and storm groups replaced by size-matched random space-time blocks. All variants get the same number of trees, the same feature-subset size, and the same 50-configuration tuning budget selected under the same storm-isolated validation.
Why not just…?
Why not just hold out whole months or years? This is the most serious threat. A coarse time-block split also keeps storms intact, and for evaluation it may come close to storm-isolated cross-validation. Our experiments report both. If time blocking matches storm isolation in relative bias, we will say so and narrow the evaluation claim. The training claim does not depend on this outcome. Holding out months does nothing to the leaky out-of-bag estimate or to the correlation between trees inside the forest. The deciding comparison is storm-bootstrap against pixel-bootstrap at matched compute, both scored on the deployment period.
Why not just use IBCV? IBCV was designed for static 2D fields such as house prices and bird presence. A moving storm can carry correlated pixels far beyond any fixed buffer within a few hours. A buffer wide enough to contain the storm also discards most of the training data, which causes the pessimistic covariate-shift bias the geospatial papers document. The direct comparison on deployment error will settle whether storm-defined groups beat distance-defined buffers.
Why not just drop latitude, longitude, and time from the features? Removing coordinates removes the most obvious shortcut. The interpolated coarse fields remain storm-specific fingerprints, however, so leakage through near-duplicate features persists. We run this as an ablation and expect it to close only part of the gap between leaky and honest error.
Why not just use a deep downscaler? Deep models leak through the same storm-sharing splits, and we audit our convolutional baseline under all splitting schemes. Trees remain worth fixing because they are cheap, fast at inference, and widely used for station and multi-source fusion. If the deep model wins on the 99th-percentile tail even under honest splits, we will report it. The leakage findings stand regardless of which model class wins.
Decisive Checks
The proposal succeeds if three conditions hold. First, pixel-level cross-validation and out-of-bag estimates are substantially optimistic against the deployment error, while storm-grouped out-of-bag error falls within the IBCV-level 9% band. Second, storm-bootstrap forests show lower measured between-tree correlation $\rho$ and equal or lower deployment error on heavy-rain events. Third, the gain disappears when storm groups are replaced by random blocks of the same size.
If storm-bootstrap and pixel-bootstrap forests reach the same deployment error, the training contribution is refuted and the work reduces to an evaluation audit. That outcome would still be useful, given that HR-Extreme and WassDiff report their numbers without event-isolated splits.
References
- Examining the challenges of blood pressure estimation via photoplethysmogram
- Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
- Cross-validation for Geospatial Data: Estimating Generalization Performance in Geostatistical Problems
- Model Evaluation for Geospatial Problems
- Diagnosing added value of convection-permitting regional models using precipitation event identification and tracking
- HR-Extreme: A High-Resolution Dataset for Extreme Weather Forecasting
- Snow Depth Fusion Based on Machine Learning Methods for the Northern Hemisphere
- Downscaling Extreme Precipitation With Wasserstein Regularized Diffusion