Rethinking Perceptual Similarity for Remote Sensing: Multi-Layer Attention Scoring and Human Preference Benchmarking

Eli Underwood, Marc Bosch, Neil Joshi, Jeffrey Wen, Myron BrownarXiv 2026 · arXiv

TL;DR

Researchers have developed Remote Sensing Perceptual Distance (RSPD), a new metric that aligns with human judgment for evaluating generated satellite imagery. By leveraging multi-layer attention scores from a domain-specific foundation model (RemoteCLIP), RSPD significantly outperforms standard metrics in identifying critical failures like misregistration and semantic errors across tasks like super-resolution and SAR-to-optical translation.

What Problem Does This Paper Address?

In the world of natural images, a "realistic" looking cat is usually a good cat. In Remote Sensing (RS), however, a "realistic" looking image can be a disaster. A generative model might produce a sharp, plausible-looking satellite tile that has shifted a road by ten meters or hallucinated a building where there is only forest.

Traditional metrics like PSNR and SSIM are too sensitive to pixel-perfect alignment, while modern learned metrics like LPIPS (trained on natural images) often fail to capture the specific geospatial and semantic nuances of overhead imagery. Furthermore, previous studies found that standard metrics often agree with human preference at roughly the level of a coin flip. There is a clear need for a metric that understands what makes an RS image "correct" beyond just looking pretty.

Methods

The authors propose a two-stage approach to build a more robust metric:

  1. Aligned Attention Score (AAS): Instead of just comparing the final "summary" vectors (embeddings) of two images, the authors look inside the transformer blocks of RemoteCLIP. They calculate the AAS by swapping the "keys" and "values" between a reference and a candidate image. This measures how well the attention patterns correspond across the two images.
  2. Layer Discovery & Calibration: Not all layers are created equal. Through a "controlled perturbation ladder," the authors discovered that shallow layers are better at catching low-level issues like blur and noise, while deeper layers are sensitive to semantic changes (like a stadium being inserted into a field).
  3. RSPD Metric: RSPD is the average of seven independently calibrated layer distances. Crucially, this metric was not tuned on human-preference labels; it was calibrated using synthetic perturbations to ensure it remains a general-purpose tool.
Figure 1. Overview of the controlled perturbation benchmark. Background scenes are drawn from RSICD [14] and semantic insertions are generated with HarmoniDiff-RS [29]. Nonsemantic families are applied to fixed RS backgrounds with progressively stronger severities calibrated by family-specific MS-SSIM targets, while the semantic examples stage increasing object insertions under controlled area progression, yielding comparable within-family ladders without forcing different artifact types to share one pixel-space notion of severity.

Datasets

The paper introduces several high-quality benchmarks to the community:

  • Controlled Perturbation Ladder: 100 high-resolution scenes from RSICD subjected to 11 types of artifacts (blur, noise, haze, misregistration, semantic insertions, etc.) at five levels of severity.
  • Human Preference Benchmark: A massive collection of 1,500 triplets. Each triplet presents a reference image and two candidates, with human annotators choosing the one that best matches the reference. This spans five tasks: synthetic artifacts, airborne super-resolution (SR), spaceborne SR, SAR-to-optical translation, and timeseries controls.
  • Ordered Misregistration Benchmark: 300 triplets specifically designed to test a metric's sensitivity to global shifts, where the "correct" answer is mathematically determined by the magnitude of the shift.
Figure 5. Representative reference-conditioned 2AFC triplets in the blind annotation layout: candidate A left, reference center, can- didate B right.

Key Insights and Ideas

  • Depth Matters: The "aha moment" of the paper is the visualization of layer specialization. Figure 2 shows that different layers "peak" in sensitivity for different artifacts. Using only the final embedding (the standard approach for CLIP-based metrics) discards vital spatial evidence needed to detect scene shifts or texture copies.
  • Attention vs. Cosine Similarity: Comparing intermediate-layer attention is significantly more informative than simple cosine distance between final embeddings. As shown in Figure 4, the final embedding often stays "flat" across moderate levels of corruption, whereas AAS responds early and monotonically.
  • Zero-Shot Human Alignment: RSPD achieves high correlation with human judgment without being trained on human labels. This suggests that the internal representations of RS foundation models already contain the necessary "knowledge" of perceptual quality; we just need the right operator (AAS) to extract it.
Comparison showing that intermediate-layer AAS is more sensitive to artifacts than final-embedding cosine distance.

Key Results

RSPD demonstrates superior performance across the board in the reference-conditioned 2AFC task:

  • Average Accuracy: RSPD achieved 89.00% accuracy across all datasets, significantly higher than the RemoteCLIP final-embedding baseline (55.07%) and generic learned metrics like LPIPS (58.27%).
  • Downstream Tasks: On realistic tasks like SAR-to-optical translation, RSPD reached 96.00% agreement with humans, outperforming even MS-SSIM (95.00%), which is typically a very strong baseline for co-registered imagery.
  • Misregistration: In the stress test for global shifts, RSPD scored 94.00%, while MS-SSIM struggled at 62.33%, proving RSPD's robustness to the alignment issues that plague traditional metrics.

Limitations and Future Work

  • RGB Focus: The current benchmark and metric primarily focus on RGB visual fidelity. Future work is needed to extend these insights to multispectral, hyperspectral, and raw SAR data.
  • Backbone Dependency: While the framework is "backbone-agnostic," its performance relies on the quality of the underlying foundation model. As stronger RS models emerge, the layer-selection process will need to be repeated.
  • Non-Exhaustive Artifacts: The perturbation families used for calibration are representative but not exhaustive of all possible generative model failures.

The authors plan to release the benchmark, codebase, and annotations to serve as a diagnostic platform for the remote sensing community.

Summary updated

Abstract

Perceptual similarity metrics for natural images transfer poorly to remote sensing (RS), where severe geospatial and semantic failures can remain visually plausible while still being decisively wrong for downstream use. To address this, we develop a novel human-aligned perceptual quality metric for generated RS imagery and evaluate it across several generation tasks. We first conduct a controlled perturbation study where we progressively increase the severity of common synthetic perturbations. Using the layer-wise Aligned Attention Score (AAS) responses in RemoteCLIP, we find that different RS artifacts are best captured at different depths. Based on these findings, we introduce Remote Sensing Perceptual Distance (RSPD), a pairwise metric that averages seven independently calibrated layer distances without tuning on human-preference labels. To evaluate our metric on realistic RS generation tasks, we construct a blind, reference-conditioned two-alternative forced-choice (2AFC) benchmark with five 300-triplet subsets spanning synthetic artifacts, airborne and spaceborne super-resolution, SAR-to-optical translation, and timeseries controls. We also construct a 300-triplet ordered misregistration benchmark with deterministic labels. Using the same RemoteCLIP checkpoint, RSPD substantially outperforms simple cosine distance with the final global embedding and is competitive with the strongest generic baselines across the realistic RS generation tasks.
BibTeX
@misc{underwood2026rethinking,
  author = {Eli Underwood and Marc Bosch and Neil Joshi and Jeffrey Wen and Myron Brown},
  title = {{Rethinking Perceptual Similarity for Remote Sensing: Multi-Layer Attention Scoring and Human Preference Benchmarking}},
  year = {2026},
  eprint = {2611.21816},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21816},
}
Figure