Block-Held-Out Uncertainty for Kernel-Regression Sensor Fusion

Many operational maps of physical quantities, such as satellite soil moisture corrected against ground stations or PM2.5 fused from monitors and satellite retrievals, are produced by a generalized regression neural network (GRNN). A GRNN is a kernel-weighted average: it stores every training example and predicts a new point's value as the average of stored targets, weighted by closeness. These pipelines report a single cross-validated error for the whole map and no per-location uncertainty. Adding that uncertainty is hard for a second reason: nobody observes the true uncertainty at a location, and ground stations are too sparse to set some aside for calibration without making the map worse.

The proposal is to attach calibrated, location-specific error bars to an existing GRNN predictor, using only the sums it already computes. Because a GRNN prediction is a ratio of two sums over stored examples, removing any group of examples, such as a whole station or a whole region, only requires subtracting their terms. The prediction the model would have made without that group is therefore exact and costs nothing to compute. We use these exact held-out predictions in three ways: to estimate local error, to calibrate prediction intervals with a finite-sample coverage argument without spending any stations on calibration, and to decide where the model has no support at all. The evaluation is built around the fact that true uncertainty is unobservable.

Core Claim

For a fixed GRNN predictor, exact block-held-out residuals are a better basis for its uncertainty than the two things a practitioner would reach for first. The first is the kernel conditional variance, the second-moment trick that estimates target spread around the local mean. The second is a generic post-hoc method that holds out stations. Our claim is that the block-held-out residuals give intervals that hold their stated coverage region by region, not just on average, under sparse and spatially clustered ground truth.

The claim is about the fixed deployed predictor. We do not propose replacing it with a different probabilistic model. Nothing in the file combines GRNN with any uncertainty formalism. The applied GRNN papers, including Quality Improvement of Satellite Soil Moisture Products and Point-surface fusion of station measurements and satellite observations for mapping PM2.5, report only R and RMSE-type scores. Those papers motivate the need. They are not competing methods.

Mechanism

Write $w_i(x) = \exp(-\|x-x_i\|^2/2\sigma^2)$ for the kernel weight of stored example $i$ at query $x$. The GRNN prediction is $\hat y(x) = \sum_i w_i y_i / \sum_i w_i$. For a block $B$ of examples (for instance all days from one station, or all stations in a spatial tile), the block-held-out prediction is

$$\hat y_{-B}(x) = \frac{\sum_{i\notin B} w_i(x)\, y_i}{\sum_{i \notin B} w_i(x)},$$

which is exact, because GRNN has no fitted weights to re-optimize. This gives every training example a residual $e_i = y_i - \hat y_{-B(i)}(x_i)$ computed as if its whole block had never been seen. We also use these residuals to choose $\sigma$.

The local uncertainty scale combines two quantities. The first is the kernel-smoothed held-out squared residual, which estimates total risk (noise plus local bias) rather than noise alone. The second is the effective number of contributing examples, $n_{\mathrm{eff}}(x) = (\sum_i w_i)^2/\sum_i w_i^2$:

$$u(x) = \sqrt{\frac{\sum_i w_i(x)\, e_i^2}{\sum_i w_i(x)}\left(1 + \frac{1}{n_{\mathrm{eff}}(x)}\right)}.$$

The factor $1+1/n_{\mathrm{eff}}$ is the variance inflation of a weighted average of $n_{\mathrm{eff}}$ independent noisy observations. Intervals take the form $\hat y(x) \pm q\,u(x)$, where $q$ is the $(1-\alpha)$ quantile of the normalized held-out scores $|e_i|/u_{-B(i)}(x_i)$. This is a locally scaled jackknife+-style construction, so no stations are consumed by a separate calibration split.

Far from all data, $n_{\mathrm{eff}}$ tends toward one and the smoothed residuals simply echo the nearest stations. The scale $u$ alone therefore cannot flag extrapolation. We add a support gate: the log denominator $\log \sum_i w_i(x)$, computed with log-sum-exp so it stays numerically stable, is compared against the distribution of block-held-out log denominators at training points. The model abstains below a calibrated quantile.

This uses the discarded GRNN denominator as a density signal. Density Uncertainty Layers achieves the same effect in deep networks only by training auxiliary energy models, and it shows why the effect is needed: variational methods become confident in empty gaps between data clusters. We borrow its criterion, not its machinery.

Research Program

The work has three parts.

The first part is the estimator above, implemented as a drop-in wrapper around a stored GRNN. Block definitions are chosen per domain: station, spatial tile, or month.

The second part is an evaluation protocol for settings where true uncertainty is unobservable. We build semi-synthetic benchmarks that keep the real station geometry and covariates of the soil moisture fusion data. A GRNN fitted to the real data serves as the true mean, and targets are resampled with prescribed noise fields: homoscedastic, heteroscedastic tied to vegetation water content, and skewed. The true conditional distribution at every location is then known, so pointwise interval accuracy can be measured directly.

This design follows the lesson of Uncertainty Quantification Metrics for Deep Regression. That paper found Spearman correlation between uncertainty and error to be unstable and to converge toward zero, while CDF-based calibration error was the most stable metric. It also follows Uncertain of uncertainties?, which showed that global miscalibration area can look perfect while regional over- and under-confidence cancel, and that Spearman jumped from 0.05 to 0.65 just by changing the test mix. We therefore report calibration per spatial region and per support stratum, never only globally, and demote Spearman to a secondary diagnostic.

The third part is a risk-decomposition analysis. Using the pointwise decomposition of Uncertainty Quantification for Regression using Proper Scoring Rules, which separates total risk from Bayes and excess risk and argues that total risk suits selective prediction while excess risk suits shift detection, we test whether $u$ tracks total risk and whether the support gate tracks excess risk on semi-synthetic data where both are known. That paper assumes Gaussian predictive distributions for deep networks and Gaussian processes. Our estimator is distribution-free and does not refit anything, which is the main contrast.

Baselines

The naive reference point is the GRNN with a single global interval, $\hat y \pm q\cdot$RMSE, calibrated from ordinary 10-fold cross-validation. This is effectively what current fusion papers imply. The strongest reported accuracy for the underlying predictor is from Quality Improvement of Satellite Soil Moisture Products: a 10-fold cross-validated R of about 0.9 (0.87 in the results text) and ubRMSE of 0.044 cm³/cm³ on the western U.S. fusion data. No reported result was found on Lacuna for any uncertainty or coverage metric on this dataset. Our intervals must therefore beat the naive global interval on regional coverage and CRPS. They must also avoid degrading that point accuracy, which by construction they do not touch.

The second baseline is the kernel conditional variance: add a summation unit for $\sum_i w_i y_i^2$ and form the conditional variance. This is the obvious "free" GRNN uncertainty, and it isolates our key ingredient, held-out residuals versus in-sample spread. It is given the same $\sigma$ and the same final conformal quantile step, so the only difference is the scale function.

The third baseline is split conformal with a learned residual scale. A fraction of stations is held out, a GRNN is fitted to absolute residuals on the held-out set, and normalized conformal quantiles are computed. It gets the same total stations. Beating it means equal coverage with narrower regional intervals, because we spend no stations on calibration.

The fourth baseline is the strongest deep uncertainty method in the file. We use Monte Carlo dropout as in Dropout-based Active Learning for Regression, with T = 25 passes, and deep ensembles via Torch-Uncertainty. Each gets the same features, the same spatial-block tuning splits, and a tuning budget matched in wall-clock time to the GRNN pipeline. Each also gets the same conformal recalibration, so we compare scale functions rather than raw calibration.

The fifth baseline is a Gaussian process with marginal-likelihood hyperparameters, used both as a replacement predictor and as a variance source for the GRNN residuals.

Ablations remove each part of our estimator separately: the $n_{\mathrm{eff}}$ factor, the support gate, and block holdout in favor of single-point holdout.

Why not just…?

Why not just use the kernel conditional variance the GRNN can already compute? It measures the spread of stored targets around the local mean using those same targets, so it ignores the bias of the local average. In sparse regions it is computed from a handful of examples, which should make it overconfident exactly where stations are missing. Baseline two tests this directly. If it matches our estimator on regional calibration error in the semi-synthetic suite, the held-out machinery is unnecessary and we will say so.

Why not just fit a Gaussian process instead? This is the most serious threat. A Gaussian process has closed-form posterior variance and closed-form leave-one-out, and on a few hundred stations it may simply be better. Two reasons keep it from settling the question. It is a different predictor from the operational GRNN, and its standard variance depends only on input geometry under constant noise, so heteroscedastic noise such as vegetation-dependent retrieval error requires extra optimization. We run the comparison in both roles. If a Gaussian process with block cross-validation matches both our accuracy and our regional calibration, the contribution reduces to the evaluation protocol and the retrofitting argument for existing GRNN pipelines. We will report that honestly.

Why not just use deep ensembles or MC dropout with conformal recalibration? Conformal recalibration fixes average coverage but not where the intervals are wrong. Density Uncertainty Layers documents confidence collapse in data gaps for approximate Bayesian networks. Torch-Uncertainty reports that deep ensembles on MUAD reached 1.58% ECE against 0.51% for a single model, and that jointly fitting mean and variance can hurt the mean. Whether any of this transfers to tabular geophysical regression is an empirical question, and the regional-coverage comparison under spatial holdout is designed to answer it.

Why not just hold out stations for split conformal? When ground truth is a few hundred stations, every station spent on calibration is one the map no longer uses. Block holdout gives the same held-out logic for free. Baseline three measures the price of splitting as interval width at equal coverage and as ubRMSE lost.

Proposed Experiments

The primary testbed is the western U.S. soil moisture fusion data from Quality Improvement of Satellite Soil Moisture Products. It combines SMAP L3 retrievals, surface temperature, land cover, vegetation water content, and location with SCAN, SNOTEL, SoilSCAPE, and USCRN stations and PBO H2O reflectometry.

We replace random 10-fold cross-validation, which leaks information between neighboring stations, with spatial-tile holdout. We expect this to lower the reported R, and we will quantify by how much.

Real-data metrics are CRPS, calibration error, and interval coverage and width per held-out tile, stratified by support level. We also report the prediction-rejection ratio for selective prediction, and a secondary check at sites where in-situ and GNSS-R estimates coexist. There, their disagreement serves as an imperfect reference for local noise. It is treated as ambiguous ground truth in the spirit of Conformal prediction under ambiguous ground truth, which showed that calibrating against a single aggregated label under-covered by about 10 points.

The semi-synthetic suite built on the same station geometry provides the decisive test. Our estimator should achieve lower per-region calibration error than all baselines. Its excess-risk signal from the support gate should rise with true excess risk. Removing block holdout should produce measurable overconfidence in sparse tiles.

UCI regression benchmarks from the proper-scoring-rules and dropout papers check that the result is not specific to geophysics. The Air Quality stream from Estimating regression errors without ground truth values, where linear segment ensembles reached a drift-detection F1 of about 0.74, tests whether the support gate flags temporal shift with no labels.

The proposal fails if the kernel conditional variance or a Gaussian process matches our regional calibration on the semi-synthetic suite. It also fails if block-held-out calibration loses coverage under spatial holdout by more than split conformal does.