Models that map satellite imagery to land cover, canopy height, or air quality are trained against very sparse ground truth, such as a few dozen air-quality stations or scattered LiDAR footprints. They are then deployed on images that are often corrupted by clouds, sensor noise, or blur. The standard safety valve is selective prediction: the model abstains when it is unsure. That valve fails under exactly the corruptions that matter. A large audit of frozen geospatial foundation models found that their internal representations barely change when inputs are corrupted, so the models stay confident while accuracy collapses and confidence-based abstention becomes useless (Beyond Accuracy).
We propose to take the abstention decision away from the network's confidence and give it to a simple physical test. Every clean optical pixel should be approximately a non-negative, sum-to-one mixture of a small set of pure material spectra (vegetation, soil, water, built surfaces). A clouded, saturated, or noise-dominated pixel falls outside the convex hull of those spectra. We fit the spectra from unlabeled imagery with multiplicative updates. Multiplicative updates are an optimizer that keeps parameters non-negative by rescaling them rather than clipping. At test time, we measure how badly each pixel fails to be reconstructed inside that hull, and abstain when it fails. The test needs no labels, works with any frozen predictor, and spends the scarce ground truth only on setting one threshold.
Core Claim
The claim is that the reconstruction residual of a simplex-constrained non-negative factorization detects the input corruptions that make geospatial predictors "confidently wrong." Gating a predictor on this residual, jointly with its own confidence, should recover a usable risk–coverage trade-off under shift where confidence alone collapses.
The specific structural bet is the sum-to-one constraint. An unconstrained linear subspace, such as principal component analysis, can represent brightened or offset versions of clean spectra, so a haze or cloud pixel can reconstruct well. A convex hull cannot represent anything brighter than its brightest endmember. If this bet is wrong, and a plain PCA residual works as well, the contribution shrinks to "input-space residuals beat embedding confidence." We will report that outcome explicitly.
The three directions play different roles. Multiplicative updates are the solver. Selective prediction is the output we evaluate. Sparse ground truth is the constraint that makes a label-free selector valuable.
Relation to Prior Work
Three pieces of prior work motivate the proposal. Beyond Accuracy documents the failure we target: mean accuracy falls from 0.84 to 0.15 under severe clouds, and calibration error reaches 0.7. That study only diagnoses the problem and proposes no remedy. The canopy-height work of Lang et al. and its Bayesian companion (Global canopy height estimation with GEDI LIDAR waveforms and Bayesian deep learning) show that uncertainty-based rejection helps on sparse LiDAR supervision. Their approach needs ten networks, and their uncertainty is learned from the same biased labels. Bui et al. predict a variance for Sentinel-2 canopy height but never turn it into a formal abstention policy.
Four pieces supply the method. The linear mixing model, with its non-negativity and sum-to-one constraints (Decomposing Mixed Pixels via Spectral Unmixing; Dobigeon et al.), gives the physical hull. Unmixing work inverts every pixel and never abstains; we use the inversion's failure as the signal. Dhillon and Sra's Bregman-divergence NMF and NNEinFact provide multiplicative updates for general divergences and missing-data masks. We reuse them, including masks for cloud-flagged training pixels. NNEinFact produces point estimates only and has no notion of abstention. Selection with Guaranteed Risk and Conformalized Selective Regression supply the calibration machinery for thresholds.
The closest multiplicative-update work that uses the word "selective" is SMRMF. It prunes graph neighborhoods during unsupervised factorization. It never abstains on test predictions, so it answers a different question.
Research Program
We first fit an endmember matrix $W$ with $K$ non-negative columns from unlabeled, nominally clean imagery of the deployment region, using β-divergence multiplicative updates. Sum-to-one is imposed with the standard augmented-row trick: we append a constant row to the data and to $W$.
For a new pixel $y$, we hold $W$ fixed and solve for abundances $a \ge 0$ with $\mathbf{1}^\top a = 1$, again by multiplicative updates. This is a small per-pixel problem that vectorizes over an image. The residual score is
$$s(y) = \min_{a \ge 0,\ \mathbf{1}^\top a = 1} D_\beta(y \,\|\, W a),$$
where $D_\beta$ is the β-divergence matched to the sensor noise. For image-level tasks, $s$ is aggregated per image as an upper quantile over pixels.
Choosing $D_\beta$ to match the noise model has a practical benefit: $s$ then behaves like a deviance statistic. Its clean-data distribution can be estimated from unlabeled pixels. The residual threshold $\tau_s$ can therefore be set as a quantile of clean unlabeled scores, with no ground truth at all.
The final selector accepts an input only if $s(y) \le \tau_s$ and the predictor's own confidence clears a second threshold. That second threshold is calibrated with SGR (classification) or CSR interval widths (regression) on the small labeled set. Scarce labels are thus spent on one scalar rather than on learning the selector.
We will extend the same construction to GEDI waveforms, which are themselves non-negative energy returns. There, $W$ holds waveform "shapes" and the residual flags anomalous returns.
Baselines
The naive reference point is no abstention: full-coverage accuracy or RMSE under each shift. This sets the floor that any selector must improve on the accepted subset.
The confidence baseline is softmax response or predicted variance with an SGR- or CSR-calibrated threshold on the same frozen predictor (Geifman and El-Yaniv; Sokol et al.). This is what practitioners use, and it is what Lehmann et al. found to break. The comparison is fair because both selectors gate the identical predictor, and both receive the identical labeled calibration set drawn from clean data. Lehmann et al. report the collapse qualitatively; no reported result found on Lacuna for a numeric selective-risk or retention-curve area on their benchmark, so we regenerate it.
The strongest existing method for sparse-label regression is the ten-member deep ensemble of Lang et al.. On 68,483 GEDI–airborne-LiDAR matched footprints, it reports that discarding the 30% most uncertain predictions lowers canopy-top RMSE from 3.6 m to 2.7 m. We will match its 70% coverage operating point. Beating it means one of two things: equal or lower RMSE at 70% coverage using one predictor plus the residual gate, or clearly lower RMSE when corruptions (cloud-contaminated or low-energy waveforms, added detector noise) are injected into the test footprints. We report compute as training and inference GPU-hours for both. If the original matched set is unavailable, we rebuild a comparable set from public GEDI L1B data and airborne LiDAR, retrain the ensemble on it, and report our retrained ensemble number alongside the published one.
We also include an embedding-distance baseline: a Mahalanobis distance on the frozen encoder features, calibrated identically. This tests directly whether representational rigidity also defeats feature-space out-of-distribution detection.
The ablations isolate the key ingredient. We replace the simplex residual with a same-rank non-negative residual without sum-to-one, then with a PCA residual, then with a small convolutional autoencoder residual. All are fit on the same unlabeled pixels with the same rank budget. A solver ablation replaces multiplicative updates with projected gradient. We expect this to change speed and stability but not the selector's quality, and we say so in advance.
Why not just…?
Why not just use the cloud mask or quality flags that come with the imagery? For clouds, this is a real competitor. We will compare against a cloud detector trained on CloudSEN12, which is part of the Lehmann benchmark. Masks, however, do not exist for Poisson-Gaussian detector noise or motion blur, which are two of the three shifts where confidence also fails. If the residual only matches the cloud mask on clouds but adds nothing on noise and blur, the method is not worth deploying, and we will say so.
Why not just use a PCA or autoencoder reconstruction error? This is the most serious threat. The ablation above settles it. Our prediction is that the sum-to-one hull flags additive brightening (haze, thin cloud) that a linear subspace absorbs. If PCA is within noise of the simplex residual on every shift and severity, we drop the physical-constraint claim. The work then becomes a more modest study of input-space residuals versus embedding confidence.
Why not just train a deep ensemble or use MC dropout? Ensembles of probes on a frozen encoder share the same rigid embedding, so they inherit the failure Lehmann et al. observed. Full ensembles, as in Lang et al., cost ten times the compute, and their epistemic uncertainty is learned from spatially biased labels. The GEDI comparison measures both accuracy and cost directly. If the ensemble wins at comparable cost on corrupted inputs, the residual gate has no case.
Why not just recalibrate with temperature scaling? Temperature scaling fitted on clean data cannot anticipate unseen corruptions. It rescales confidence but preserves its ranking, so it cannot fix abstention when the ranking itself is wrong. We include it as a variant of the confidence baseline.
Proposed Experiments
The first testbed is the Lehmann et al. shift protocol on the multispectral classification sets: m-EuroSAT and So2Sat, with RESISC45 as an RGB stress case where the hull is weaker. We apply cloud/shadow, noise, and blur at five severities to frozen encoders with linear probes. Metrics are the area under the risk–coverage curve, accuracy at 80% coverage, the fraction of corrupted inputs rejected, and the coverage lost on clean data.
The second testbed is regression on sparse ground truth. We use Sentinel-2 canopy height against GEDI in the Vietnam reserves of Bui et al., training in one reserve pair and testing in the other to create a regional shift. We also use GEDI waveforms against airborne LiDAR in the style of Lang et al. Metrics are RMSE at matched coverage, the error–coverage area used by CSR, and the ideal-point distance.
The decisive check is severity-five clouds and noise on m-EuroSAT. There, accuracy on inputs accepted by softmax response should sit near the collapsed full-coverage level, while accuracy on inputs accepted by the residual-gated selector should stay near clean accuracy, at a clean-data coverage cost of a few percent. A second decisive check is the clean-data false-rejection rate in dense forest, where nonlinear multiple scattering violates the linear mixing model. If that rate exceeds roughly 15%, the hull must be enlarged or learned per biome before the method is usable.
References
- Beyond Accuracy: Assessing Calibration of Geospatial Foundation Models and Their Sensitivity to Distribution Shifts
- Global canopy height regression and uncertainty estimation from GEDI LIDAR waveforms with deep ensembles
- Deep Regression Neural Network for Estimating Canopy Height in Vietnam's National Forests
- Joint Bayesian endmember extraction and linear unmixing for hyperspectral imagery
- Generalized Nonnegative Matrix Approximations with Bregman Divergences
- Near-Universal Multiplicative Updates for Nonnegative Einsum Factorization
- Selective Classification for Deep Neural Networks
- Conformalized Selective Regression
- Unsupervised Selective Manifold Regularized Matrix Factorization