When a disaster strikes somewhere a model has never seen, responders need two things from an automated damage map. They need a damage grade for each building, and they need to know which of those grades to trust. The usual way to judge "which tiles are usable" relies on generic image-quality measures such as sharpness, contrast, entropy or signal-to-noise. On satellite land-cover classification these measures have close to zero correlation with how well the classifier actually does IMAGE QUATY ASSESSMENT FOR VHR REMOTE SENSING IMAGE CLASSIFICATION. Meanwhile, damage models lose up to 30% of their accuracy when moved to new geographic sites Position: Mission Critical – Satellite Data is a Distinct Modality in Machine Learning. The quality question that matters is therefore not "does this image look good" but "how wrong will the model be on it, here."
This note proposes that the answer can come from the damage model itself, provided it is trained the right way. Building damage is graded on an ordered scale: undamaged, minor, major, destroyed. Suppose the model is trained with a distance-aware loss, meaning a loss that penalizes predicting "destroyed" for an undamaged building more than predicting "minor." Then the spread of its predicted distribution along that scale becomes a direct estimate of how many grades off the prediction is likely to be. We call this spread the expected ordinal distance. We test whether it works as a task-specific, label-free quality score for each tile and each region under geographic shift, and whether it beats generic quality metrics, ordinary confidence scores, and ensemble variance from regression models.
Core Claim
Training with an ordinal loss changes more than accuracy. It changes what the model's output probabilities mean. Cross-entropy treats all wrong classes as equally wrong, so its confidence says only "how likely am I exactly right." A distance-aware loss arranges probability mass along the scale, so dispersion becomes informative about error magnitude. Write $p_i(x)$ for the predicted probability of damage grade $i$ on building crop $x$, and $d(i,j)=|i-j|$ for the distance between grades. Let $\hat y(x)$ be the median of $p(x)$, which is the prediction that minimizes expected absolute grade error. The proposed score is
$$ s(x) = \sum_{i=0}^{3} p_i(x)\, d\big(\hat y(x), i\big), $$
the model's own expected number of grades it is off. The claim has three parts. On disasters held out entirely from training, $s(x)$ ranks buildings by their realized absolute grade error better than generic image-quality indices, better than cross-entropy max-probability or entropy, and at least as well as the variance of a regression ensemble at a fraction of the cost. Averaged over a region, $s$ predicts which unseen disasters the model will do badly on. Under controlled blur, noise and haze, $s$ rises gradually before accuracy collapses, whereas generic metrics only flag the collapse point IMAGE QUATY ASSESSMENT FOR VHR REMOTE SENSING IMAGE CLASSIFICATION.
The closest prior work plays three different roles. Adaptable image quality assessment using meta-reinforcement learning of task amenability learns a separate agent that scores whether an ultrasound image is usable for a downstream task. It motivates the target, which is task-specific quality, but it needs expert-labeled rewards, costly meta-reinforcement learning, and binary or segmentation tasks. Our score is a free byproduct of the task model, is ordinal, and is tested under geographic shift. The ordinal losses themselves supply the method. Ordinal Log-Loss penalizes $-\log(1-p_i)$ weighted by $d(y,i)^{\alpha}$ and improved off-by-one accuracy on Amazon reviews from 90.9% to 92.5% A Simple Log-based Loss Function for Ordinal Text Classification. Earth-mover-style losses cut numeric errors in multimodal language models by over half Advancing Sequential Numerical Prediction in Autoregressive Models. Neither work examines what these losses do to uncertainty on shifted image inputs.
One paper constrains the design. Reducing Supervision Uncertainty Induces Model Miscalibration shows that reshaping supervision targets can make models overconfident. Ordinal Log-Loss aggressively suppresses mass on distant grades, so it may make $s$ collapse toward zero. That is exactly why we compare it against an earth-mover loss, which keeps mass on neighboring grades. We treat calibration of $s$ as a measured outcome, not an assumption.
Research Program
The main testbed is xBD, which has 850,736 building annotations over 19 disasters in 15 countries, labeled on the four-level Joint Damage Scale xBD: A Dataset for Assessing Building Damage from Satellite Imagery. Its standard 80/10/10 split mixes locations, which inflates scores through spatial leakage Deploying an Artificial Intelligence Application to Detect Flood from Sentinel 1 Data. We therefore build leave-one-disaster-out folds, grouped so that held-out events differ in country and hazard type.
To isolate damage grading from building localization, the model classifies pre- and post-event crops around ground-truth footprints using a Siamese encoder. The encoder is initialized with the DINO-plus-reconstruction self-supervised recipe of Self-Supervised Learning for Building Damage Assessment from Large-scale xBD Satellite Imagery Benchmark Datasets, and the backbone and schedule are identical across all losses. We train with cross-entropy, focal cross-entropy as in Pyramid Pooling Module-Based Semi-Siamese Network: A Benchmark Model for Assessing Building Damage from xBD Satellite Imagery Datasets, Ordinal Log-Loss with $\alpha\in\{1,1.5,2\}$, a squared earth-mover loss on cumulative probabilities, and scalar regression with smooth-L1. Because adjacent grades such as minor and major are the dominant confusion in xBD, class-frequency reweighting is applied identically to every loss. We report how it interacts with distance weighting.
The second arm uses PovertyMap-WILDS, which predicts a continuous wealth index across country folds WILDS: A Benchmark of in-the-Wild Distribution Shifts. We bin the index into quantile bins, train with the same ordinal losses, and use $s$ to gate pseudo-labels on unlabeled target countries. This directly tests whether ordinal discretization makes self-training usable where Extending the WILDS Benchmark for Unsupervised Adaptation found adaptation methods gave no gain on regression. The third arm is a degradation sweep: synthetic blur, Gaussian noise and haze at graded strengths applied to xBD post-event imagery, following the protocol logic of the Zhengzhou study.
Baselines
The naive reference point is a cross-entropy model paired with no quality filtering at all, so every building's prediction is trusted equally. Its own maximum softmax probability and predictive entropy serve as the "free" confidence scores. This is the default in the xBD baseline, which reached a weighted F1 of 0.2654 xBD: A Dataset for Assessing Building Damage from Satellite Imagery. It is the right floor because it is what practitioners already get for free. Beating it means higher rank correlation between score and absolute grade error on held-out disasters, and a lower area under the risk-coverage curve, where risk is mean absolute grade error over the retained buildings.
The generic image-quality baseline computes standard deviation, average gradient, entropy, contrast, homogeneity, and a no-reference sharpness proxy on each post-event crop. These are the indices tested in IMAGE QUATY ASSESSMENT FOR VHR REMOTE SENSING IMAGE CLASSIFICATION. We also add a small learned regressor that combines them, trained on source disasters to predict error. That study reports near-zero $R^2$ on land cover but gives no numeric values, and no reported result found on Lacuna for these indices as error predictors on xBD. Our score must beat the learned combination, not just the individual indices, on held-out disasters.
The strongest competitor for the uncertainty role is a five-member ensemble of smooth-L1 regressors, scored by the variance of member predictions, plus the same ensemble for cross-entropy scored by disagreement. Ensembles are the standard strong uncertainty method and need no custom loss. We report them at their true fivefold training and inference cost, and against a five-member ordinal ensemble so that the loss comparison is also made at matched cost. A single ordinal model counts as beating the regression ensemble if its rank correlation with error is within noise of, or above, the ensemble's at one-fifth the compute.
For damage accuracy itself, the reference numbers are PPM-SSNet's per-class F1 of 0.90, 0.41, 0.65 and 0.70 for undamaged, minor, major and destroyed Pyramid Pooling Module-Based Semi-Siamese Network: A Benchmark Model for Assessing Building Damage from xBD Satellite Imagery Datasets, and the self-supervised model's F1 of 0.636 at 20% labels Self-Supervised Learning for Building Damage Assessment from Large-scale xBD Satellite Imagery Benchmark Datasets. These were measured on the random split, so they are context rather than direct targets. We rerun the focal-loss recipe on our disaster-held-out folds. On PovertyMap the reference is ERM's out-of-distribution Pearson correlation of 0.45, against 0.60 in-distribution WILDS: A Benchmark of in-the-Wild Distribution Shifts. Score-gated pseudo-labeling beats it only if it exceeds 0.45 averaged across folds, with confidence intervals over seeds.
Two ablations isolate the key ingredient. The first computes the same expected-distance score on the cross-entropy model's softmax. This separates the effect of the loss from the effect of the score's formula. The second uses entropy instead of expected distance on the ordinal model, which separates the score's formula from the loss. The central claim holds only if the ordinal-trained, expected-distance combination beats both.
Why not just…?
Why not just regress the damage grade with MSE and use the prediction's residual scale? A single regressor outputs one number and has no native spread, so it needs an ensemble or a variance head to produce any quality signal. Regression also left adaptation methods helpless on PovertyMap because they rely on class probabilities Extending the WILDS Benchmark for Unsupervised Adaptation. This is a real threat for the uncertainty role, which is why the regression ensemble is our strongest baseline. If it matches the single ordinal model at matched compute, we will report that the loss contributes accuracy but not a better quality score.
Why not just filter tiles with standard quality metrics or cloud masks? The direct evidence on very-high-resolution imagery is that these indices do not track classification accuracy except at catastrophic degradation IMAGE QUATY ASSESSMENT FOR VHR REMOTE SENSING IMAGE CLASSIFICATION. The broader restoration literature shows pixel-faithful outputs can even hurt detectors Single Image Deraining: A Comprehensive Benchmark Analysis. More fundamentally, the dominant failure under geographic shift is unfamiliar roofs, vegetation and building styles, not image degradation, and no image-level index sees that. The degradation sweep and the region-level ranking test will show this concretely.
Why not just learn a separate amenability predictor, as in meta-reinforcement learning? That approach needs labeled rewards and a costly training loop Adaptable image quality assessment using meta-reinforcement learning of task amenability. A predictor trained on source disasters also faces the same geographic shift as the task model. Our learned-combination baseline is a cheap stand-in for this family. If a supervised error predictor trained on source folds transfers better than $s$, that is a legitimate negative result for our claim.
Why not just use domain-invariant training like CORAL or DANN? These methods failed to beat plain training on most WILDS datasets WILDS: A Benchmark of in-the-Wild Distribution Shifts. In any case they target accuracy, not knowing when to abstain, so they are complementary. We include CORAL on one fold as a sanity check rather than a main comparison.
Decisive Checks
The first check is building-level. On each held-out disaster, we compute the Spearman correlation between the score and the realized absolute grade error, along with the risk-coverage area, for every loss-and-score pair. The claim survives only if the ordinal-trained expected-distance score beats cross-entropy confidence and the learned generic-quality combination in a majority of folds, and ties or beats the regression ensemble at lower cost.
The second check is region-level. We rank held-out disasters by mean $s$ and correlate that ranking with realized mean absolute error. This is the operationally important test, because it tells a responder before any labels exist whether a new event's map is trustworthy.
The third check is the degradation sweep. We plot $s$, generic indices, and accuracy against degradation strength, and check whether $s$ rises before accuracy falls. The fourth check is calibration. We compare predicted $s$ with observed mean error in bins, separately for Ordinal Log-Loss and the earth-mover loss, to test the overconfidence risk raised by Reducing Supervision Uncertainty Induces Model Miscalibration. The fifth check is score-gated pseudo-labeling on binned PovertyMap against the 0.45 out-of-distribution Pearson reference.
If the cross-entropy ablation scores as well as the ordinal model, the finding reduces to "expected distance is a good way to read any classifier," which is still useful but a smaller claim. If ensemble variance dominates, the proposal's quality-score thesis is refuted, and we will say so.
References
- IMAGE QUATY ASSESSMENT FOR VHR REMOTE SENSING IMAGE CLASSIFICATION
- Position: Mission Critical – Satellite Data is a Distinct Modality in Machine Learning
- Adaptable Image Quality Assessment Using Meta-Reinforcement Learning of Task Amenability
- A Simple Log-based Loss Function for Ordinal Text Classification
- Advancing Sequential Numerical Prediction in Autoregressive Models
- Reducing Supervision Uncertainty Induces Model Miscalibration
- xBD: A Dataset for Assessing Building Damage from Satellite Imagery
- Deploying an Artificial Intelligence Application to Detect Flood from Sentinel 1 Data
- Self-supervised Learning for Building Damage Assessment from Large-Scale xBD Satellite Imagery Benchmark Datasets
- Pyramid Pooling Module-Based Semi-Siamese Network: A Benchmark Model for Assessing Building Damage from xBD Satellite Imagery Datasets
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Extending the WILDS Benchmark for Unsupervised Adaptation
- Single Image Deraining: A Comprehensive Benchmark Analysis