STARS: Stage-Adaptive Responsibility Separation for Shadow Removal in Remote Sensing Images

Gaoliang Song, Chengjie Li, Pengju Liu, Chenchen Chang, Jinhao You, Hongzhi ZhangarXiv 2026 · arXiv

TL;DR

STARS (Stage-Adaptive Responsibility Separation) is a new framework that solves a persistent problem in shadow removal: the accidental "brightness shift" in non-shadowed regions. By decoupling the restoration process into a three-stage cascade—where the final stage is explicitly tasked with preserving non-shadow areas—and leveraging foundation model priors via a novel adapter, the authors achieve state-of-the-art results across both remote sensing and natural image datasets.

What Problem Does This Paper Address?

In remote sensing, shadows aren't just an aesthetic issue; they hide critical information for object detection and land-use classification. Most modern shadow removal models use a "mask prior"—a binary map telling the network where the shadows are.

However, the authors identify a frustrating trade-off: when a single network is trained to restore shadows, it often inadvertently "modulates" the non-shadow regions, causing a systematic shift in brightness. In remote sensing, where non-shadow regions typically occupy the vast majority of the image, even a tiny shift in these areas leads to a massive drop in overall image quality metrics (like PSNR). The fundamental challenge is that "restoring a shadow" and "leaving a non-shadow region unchanged" are conflicting tasks that compete for the same neural weights during optimization.

Methods

The authors propose a "task-decoupling" perspective realized through three main components:

  1. Stage-Adaptive Responsibility Separation (STARS): Instead of one big network, they use a three-stage cascade of "ShadowFormer" backbones.
  2. Stages 1 & 2 are fed the shadow mask ($M$) and focus on restoring the shadowed pixels.
  3. Stage 3 is fed the inverse mask ($1-M$). Its explicit responsibility is to correct any brightness shifts or artifacts introduced to the non-shadow regions by the previous stages.
  4. Feature Selection and Fusion Adapter (FSFA): To bring high-level "common sense" into the model, the authors tap into DINOv3, a large-scale vision foundation model. Because DINOv3 was trained on natural images and remote sensing data is different, the FSFA calibrates these features using per-image statistics and learnable weights to select the most useful layers (balancing mid-level details with deep semantics).
  5. Region-Specific Loss ($\mathcal{L}_{ns}$): In the final stage, the model uses a specific L1 loss that only calculates gradients for non-shadow pixels. This ensures the optimization signal for the final stage is purely focused on its assigned responsibility: identity mapping for non-shadow regions.

Datasets

The model was evaluated on three challenging benchmarks:

  • UAV-SC+: A specialized remote sensing dataset captured by unmanned aerial vehicles.
  • ISTD+: A classic, color-corrected benchmark for shadow removal in natural, everyday scenes.
  • WSRD+ 2026: The benchmark used for the CVPR 2026 NTIRE Image Shadow Removal Challenge, representing the current "gold standard" for the field.

Key Insights and Ideas

  • Structural Decoupling over Joint Optimization: The most important insight is that regional heterogeneity (shadow vs. non-shadow) requires structural separation. You can't just ask one set of weights to do two opposite things at once; you have to assign those tasks to different stages of a pipeline.
  • The Mask as a Gate, Not Just an Input: In STARS, the mask isn't just a hint; it acts as a structural gate ($M$ vs $1-M$) that defines the "responsibility" of each stage in the cascade.
  • Foundation Models for Restoration: The paper demonstrates that even though foundation models like DINOv3 aren't trained for restoration, their internal representations of "what an object should look like" provide invaluable guidance for filling in shadowed areas, provided they are properly adapted (via FSFA).

Key Results

STARS outperformed all existing methods across all three datasets:

  • UAV-SC+: Achieved a whole-image PSNR of 26.65 dB, significantly outperforming the previous best (RASM at 24.63 dB). Crucially, it improved non-shadow region PSNR to 28.08 dB, proving it successfully mitigated the brightness shift.
  • ISTD+: Set a new state-of-the-art with a shadow-region PSNR of 38.72 dB and a non-shadow PSNR of 37.47 dB, showing the method works just as well for natural photos as it does for satellite imagery.
  • WSRD+ 2026: STARS beat the winning solution of the CVPR 2026 NTIRE challenge by 0.83 dB in overall PSNR and significantly improved perceptual metrics (SSIM and LPIPS).

Limitations and Future Work

The primary drawback of the STARS framework is its computational cost. Because it relies on a three-stage cascade and a large frozen foundation model (DINOv3) for guidance, it requires more memory and time than single-stage models.

The authors suggest that future work should focus on efficiency, perhaps by exploring parameter-sharing between the stages or developing lighter versions of the semantic guidance adapter to make the model more practical for real-time remote sensing applications.

Summary updated

Abstract

Shadows are a ubiquitous radiative attenuation phenomenon in remote sensing images and can severely affect high-resolution remote sensing interpretation tasks. In shadow removal, although introducing a mask prior provides explicit positional information for the restoration network, it also causes non-shadow regions to be modulated by the shadow prior, producing a systematic brightness shift; notably, this phenomenon is independent of whether the mask is accurate. In remote sensing images, non-shadow regions usually occupy a larger proportion, so a small shift over these regions incurs a much higher cost on whole-image metrics than an equal gain in shadow regions. Adopting a task-decoupling perspective, we therefore decompose shadow removal into two tasks, shadow-region restoration and non-shadow-region identity, within a cascaded architecture, and handle them sequentially. Based on this insight, we propose STARS, a cascaded shadow removal framework that separates regional responsibilities across stages. The method retains the advantages of the shadow mask while resolving the degradation of non-shadow regions, without requiring a complex parallel dual-branch structure. Meanwhile, to introduce large-model prior knowledge into remote sensing tasks, we further design a Feature Selection and Fusion Adapter (FSFA). We conduct extensive experiments on three benchmark datasets, UAV-SC+, ISTD+ and WSRD+ 2026, demonstrating the superiority of STARS. The framework shows that decoupling regional responsibilities can resolve the intrinsic contradiction of mask priors, providing a new direction for image restoration with region heterogeneity.
BibTeX
@misc{song2026stars,
  author = {Gaoliang Song and Chengjie Li and Pengju Liu and Chenchen Chang and Jinhao You and Hongzhi Zhang},
  title = {{STARS: Stage-Adaptive Responsibility Separation for Shadow Removal in Remote Sensing Images}},
  year = {2026},
  eprint = {2611.20698},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20698},
}
Figure