Overhead aerial imagery is a cornerstone of modern urban planning, from estimating solar potential to mapping urban heat islands. However, a persistent "silent failure" haunts these models: a model trained on the leafy streets of Chicago often stumbles when deployed over the desert sprawl of Phoenix.
In their paper, ShadowTransfer: A Geographic Transfer Benchmark for Overhead Shadow Detection, researchers from Purdue University introduce a rigorous new benchmark and diagnostic framework to understand and bridge this geographic gap.
TL;DR
The authors introduce ShadowTransfer, the first human-verified benchmark for measuring how shadow detection models transfer across different cities. They discover that geographic shift causes a class-asymmetric confidence collapse in feature space—not just pixel statistics—and propose a Spectral Information Bottleneck (SIB) module that significantly reduces the transfer gap.
What Problem Does This Paper Address?
Deep learning models for dense prediction (like segmentation) are notoriously sensitive to domain shift. In the world of overhead imagery, this manifests as geographic shift. Differences in urban morphology, climate, and even sensor characteristics between cities can cause performance to plummet.
Until now, the field lacked a standardized way to measure this. Existing datasets either lacked a geographic transfer protocol, relied on noisy automatically generated labels, or focused on ground-level imagery that doesn't match the geometry of satellite views. Without a benchmark, we couldn't diagnose where the models were failing or how to fix them.
Methods
The authors tackle this with a three-pronged approach:
- The Benchmark: They created a dataset of 4,500 human-verified shadow masks across three U.S. cities (Chicago, Miami, Phoenix) at two native resolutions (0.3m and 0.6m).
- The LOCO Protocol: They established a Leave-One-City-Out (LOCO) evaluation protocol. A model is trained on two cities and tested on the third, isolating the effect of geographic shift.
- The Diagnostic Framework: They developed three "probes" to localize the failure:
- Where: Does the failure happen in pixel space or feature space?
- How: How does the prediction distribution distort?
- Which: Which architectural components (encoder vs. decoder) carry the failure?
- The Solution (SIB): Guided by their findings, they built the Spectral Information Bottleneck (SIB). This plug-in module uses Haar wavelet decomposition to separate high-frequency shadow geometry from low-frequency city-specific textures, applying a variational information bottleneck to "forget" the domain-specific noise.
Datasets
- ShadowTransfer (New): 4,500 RGB patches (384x384) from NAIP imagery. It includes six typed shadow categories (e.g., building-cast, tree-canopy dapple).
- S-EO: An external large-scale satellite dataset used for cross-validation of the benchmark's findings.
- INRIA Building Footprints: Used to prove that the observed transfer failure patterns aren't unique to shadows but apply to other overhead dense-prediction tasks.
Key Insights and Ideas
- Pixel matching is a trap: Simple histogram matching (aligning the colors of the test city to the training cities) actually degrades performance. The failure lives in the high-level features, not the raw pixels.
- Class-Asymmetric Collapse: When a model moves to a new city, it doesn't just start hallucinating shadows everywhere. Instead, it suffers a "confidence collapse" specifically on shadow pixels, while its confidence in the background remains relatively intact.
- The Locus is Architecture-Dependent: In a surprising "aha moment," the researchers found that the failure point moves depending on the model. For ViT-based models (like DINOv3), the failure is in the decoder; for CNNs (like OGLANet), it's in the encoder.
Key Results
- The Gap is Real: Under standard training, holding out a test city costs between 3.5 and 7.8 mIoU across different architectures.
- Adaptation is Risky: Six out of eight existing domain adaptation methods actually performed worse than simple "vanilla" training on average.
- SIB Wins: The proposed SIB module achieved the smallest mean gap to the "within-city" upper bound (6.42 mIoU), performing on par with the best existing methods (like FDA) but showing more consistency across different architectures and cities.
Limitations and Future Work
- Geographic Scope: The benchmark currently covers North American urban grid patterns. Transferring to non-grid morphologies or different climatic regimes (e.g., tropical or arctic) remains an open challenge.
- Sensor Diversity: The study focuses on RGB imagery from the NAIP program. Future work could explore how these findings hold up with multispectral or SAR data.
- Causal Mechanisms: While the paper localizes where the failure happens, the exact causal link between specific urban features and feature-space distortion is still being unraveled.
By providing the code, trained models, and the LOCO protocol, the authors have given the remote sensing community a powerful new "shared instrument" to ensure that models built in one corner of the world can actually work in another.
Summary updated