Where Degradation Hurts: Task-Referenced Quality Scores for Satellite Imagery Across Regions

Satellite images arrive with haze, sensor noise, radar speckle, blur and compression artifacts, and operators must decide which acquisitions are good enough to feed a flood map or a change-detection model. Today they use quality metrics borrowed from consumer photography. Those metrics are tuned to what people notice in snapshots, such as faces and foreground objects, and they treat every pixel of an overhead tile alike. We propose a no-reference quality score for Earth observation imagery whose target is how much a given degradation will hurt a specific downstream model on that tile. The score weights local damage by how sensitive the task model is to each region, rather than by human gaze or ImageNet object classes. The central question is geographic. A score calibrated in one part of the world may not hold in another. Haze over fragmented smallholder fields may destroy a crop or flood signal that the same haze over large monocultures leaves intact.

The work has two parts. First, we build a benchmark that crosses controlled physical degradations with geographic hold-out splits. It measures whether degradation and region interact, a question that existing distribution-shift benchmarks for remote sensing do not ask. Second, we train a quality estimator that predicts the degradation-induced performance drop on unseen regions and keeps that drop separate from the drop caused by the region itself. A model's own confidence cannot make this separation, yet operators need it: a hazy tile calls for re-acquisition, while an unfamiliar region calls for adaptation.

Background and the gap

Semantic-aware image quality assessment (IQA, scoring how degraded an image is, here without a pristine reference) weights distortions by what matters in the scene. Fang et al. multiply distortion by class-specific weights from an off-the-shelf segmentation network (Perceptual Quality Assessment of Smartphone Photography). TOPIQ passes high-level semantic features down to select low-level distortion features (TOPIQ). IE-IQA pairs a trainable distortion branch with a frozen "intelligibility" branch pre-trained on recognition, and selects features by gradient sensitivity (IE-IQA). These papers supply our mechanism, but they also constrain it. All of them define relevance with natural-photograph priors. IE-IQA itself reports that fine-tuning the semantic branch on small IQA sets damages it, which leaves that branch frozen on consumer-photo semantics. The file lists exactly this reliance on web-scale object ontologies as a known failure for overhead data.

On the geographic side, the motivating evidence is strong. Rolf et al. report drops of up to 30% when natural-image methods are moved to satellite data (Position: Mission Critical). Fraccaro et al. show that random splits hide site memorization in flood detection (Deploying an AI Application to Detect Flood from Sentinel 1 Data). WILDS documents large in-distribution versus out-of-distribution gaps, such as PovertyMap correlation falling from 0.60 to 0.45 (WILDS). The file names one gap directly: these benchmarks ignore how physical degradations interact with spatial shift.

Three papers come closest to joining the two directions, and each leaves the target open.

Our proposal differs from all three. Quality is defined by the effect on a task, it is computed region by region within a tile, and it is tested for transfer across geography.

Core Claim

Take a clean tile $x_0$ with labels $y$ and a physically modeled degradation $D$. Let $m$ be a task metric such as per-tile intersection-over-union, and $f$ a fixed task model. We define the tile's task-referenced quality loss as

$$\Delta(x_0, D) = m\big(f(x_0), y\big) - m\big(f(D(x_0)), y\big).$$

The first term is the geography term: how well the model does on this region when the image is clean. $\Delta$ is the degradation term. A model's confidence on $D(x_0)$ only sees their combination.

We make three claims.

  1. $\Delta$ depends on region even when degradation type and severity are held fixed. If so, any quality metric with fixed, geography-free weights will mis-rank tiles in unseen regions.
  2. A no-reference estimator $\hat{\Delta}(D(x_0))$ transfers better to held-out regions if it pools local distortion features with weights from the task model's own spatial sensitivity rather than from segmentation classes. This is because class composition and class appearance shift across regions, as with roof materials and canopy cover in the tropics (Change Detection Methods for Remote Sensing in the Last Decade).
  3. Pairing $\hat{\Delta}$ with an estimate of the clean-image performance lets operators separate "re-acquire" from "adapt" decisions better than confidence alone.

Research Program

Benchmark. We synthesize degradations matched to each sensor on datasets that already have geographic hold-outs. For optical data these are atmospheric haze, Gaussian and striping sensor noise, defocus and motion blur, and compression. For radar these are speckle and quantization. Each comes at graded severities. Because the clean tile and its labels are known, $\Delta$ is exact for every tile, degradation and severity. We do not need human opinion scores, which is fortunate since none exist for this data.

Interaction model. We fit a mixed-effects model of $\Delta$ with degradation type, severity, region and their interaction as factors. This gives the interaction's share of explained variance with bootstrap intervals, following the statistical practice of GEO-Bench (GEO-Bench).

Estimator. We adopt IE-IQA's two-branch design with one substitution. The frozen semantic branch is a geospatial foundation model (a large network pre-trained on satellite imagery), Prithvi 2.0, instead of an ImageNet network. The semantic weighting comes from the task head's gradient sensitivity map computed on the degraded input. This extends IE-IQA's sensitivity-based selection from choosing features to choosing spatial regions. A small trainable distortion branch produces local damage features, and these are pooled with the sensitivity weights.

Clean-performance head. A second head regresses the clean-image performance $m(f(x_0), y)$ from features that the degradation branch cannot see. An orthogonality penalty between the two heads keeps one from absorbing the other.

Task model. The downstream model is Prithvi 2.0 adapted with low-rank adapters (LoRA, which trains small added weight matrices while freezing the backbone). This follows Marti Escofet et al., who found that LoRA generalizes better than full fine-tuning on geographic hold-outs (Fine-tune Smarter, Not Harder). We deliberately add no coordinate inputs, because the same paper found coordinate metadata gave negligible gains. We test that choice as a baseline rather than assume it.

Baselines

Do nothing. The naive reference is Edge Intensity with uniform pooling. The 2017 remote-sensing study recommended it for unknown degradations, and it costs nothing. Beating it on rank correlation with $\Delta$ in held-out regions is the minimum bar. No reported result was found on Lacuna for any IQA metric predicting a downstream performance drop on satellite imagery, so this baseline's number is our own first measurement.

Strongest learned IQA. We run TOPIQ and IE-IQA two ways: off the shelf, and retrained on our training-region $\Delta$ labels with the same data, epochs and hyperparameter-search budget as our estimator. These are the strongest semantic IQA models in the file. The reported reference points are on natural images: TOPIQ reaches SRCC 0.813 on PIPAL, and IE-IQA reaches PLCC 0.839 when trained on KonIQ-10k and tested on LIVEW. Those numbers do not transfer to our target. Beating either retrained model means higher Spearman correlation with $\Delta$ on held-out regions, with a bootstrap interval excluding zero difference.

Strongest failure predictor. We run Saha et al.'s logit confidence and plain predictive entropy from the same LoRA task model. On OSCD, their indicator fell from 1.0 on Montpellier (Kappa 0.57) to 0 on Dubai (Kappa 0.06), where an unsupervised fallback reached Kappa 0.18. We reproduce their OSCD protocol and also apply the indicator to our degraded variants.

Ablations. Three ablations isolate the key ingredient. The first replaces sensitivity-weighted pooling with Fang-style class-weighted pooling using a segmentation network trained on the training regions. The second uses uniform pooling. The third swaps the geospatial foundation model for an ImageNet backbone in the frozen branch. A further baseline adds coordinate embeddings in the style of SegDesicNet (SegDesicNet). This tests whether a quality model needs explicit location even though the task model does not.

Why not just…?

Why not just use the task model's own confidence? This is the most serious threat. Confidence is free, and it already tracks failure in new cities. But it measures the combined drop, so a tile that fails because it is hazy looks like a tile that fails because its buildings are unfamiliar. Our factorial benchmark settles the question directly. Within held-out regions, we measure how well each signal ranks $\Delta$ alone, separately from clean-image performance. If confidence matches our estimator on that ranking and on the re-acquire-versus-adapt decision, the dedicated estimator is unnecessary, and we will report that.

Why not just fine-tune TOPIQ or IE-IQA on satellite data? We do this in the baselines, at matched budget. The 2017 study found that learned metrics carry natural-image priors into overhead data. IE-IQA's own frozen-backbone finding suggests that fine-tuning the semantic branch on small labeled sets degrades it. The ablation that swaps backbones and weighting schemes shows which component matters.

Why not just make the task model robust with degradation augmentation, so quality stops mattering? Augmentation can shrink $\Delta$, but it cannot restore information that heavy haze or speckle destroyed. We include an augmentation-trained task model as a condition. If its $\Delta$ becomes negligible across regions, quality-based triage is moot for that task. That would be a useful negative result, and we would report it as such.

Why not just regress $\Delta$ directly from the image with any network? That is our uniform-pooling ablation. If it transfers as well as sensitivity weighting, the semantic component adds nothing and the claim fails.

Proposed Experiments

Datasets and splits.

  • Sen1Floods11 with its geographic hold-out set covers radar and optical flood segmentation. On that hold-out set, LoRA reaches 87.57% mean intersection-over-union versus 82.07% for full fine-tuning (Marti Escofet et al.), so our task model starts from a known, strong operating point.
  • OSCD covers change detection, using Saha et al.'s city splits so that the diverse-versus-localized result can be re-examined under degradation.
  • The FMoW and PovertyMap splits from WILDS add cross-country classification and regression. WILDS 2.0 also supplies unlabeled target data for an unsupervised-adaptation comparison (Extending the WILDS Benchmark for Unsupervised Adaptation).
  • Two GEO-Bench segmentation tasks test whether the findings hold beyond floods.

Metrics.

  • Spearman and Pearson correlation between $\hat{\Delta}$ and $\Delta$ on held-out regions, reported per region with worst-region values, in the spirit of WILDS worst-group evaluation.
  • The share of $\Delta$'s variance explained by the degradation-by-region interaction.
  • For the decision task, area under the curve for separating degradation-dominated from region-dominated failures, using ground truth from the counterfactual decomposition.
  • Operational value under a fixed re-acquisition budget: replace the top-$k$ tiles ranked by each method with their clean versions, and measure the task metric recovered.

Decisive checks. Four results decide the outcome.

  1. If the interaction term explains a negligible share of variance, geography-free quality metrics are adequate. The semantic and geographic weighting is then unnecessary, and the paper becomes a negative benchmark result.
  2. If sensitivity-weighted pooling does not beat class-weighted and uniform pooling on held-out regions, claim 2 fails.
  3. If task-model confidence equals our estimator at separating degradation from region failures, claim 3 fails.
  4. If our estimator's advantage holds in the worst held-out region and not merely on average, it is the kind of robustness operators actually need.