FreeDeshadow: Shadow Removal for User-Specified Objects With Multimodal Prompts

Zhiliang Wu, Jihao Gu, Kun Li, Hanyu Xuan, Hehe Fan, Meng Wang, Weisi LinarXiv 2026 · arXiv

TL;DR

FreeDeshadow introduces a new paradigm for image editing: object-aware controllable shadow removal. Instead of requiring precise shadow masks or removing all shadows globally, it allows users to specify a target object via multimodal prompts (like a simple dot, a line, or a descriptive sentence) and removes only that object’s cast shadow while keeping the rest of the scene intact.

What Problem Does This Paper Address?

Shadow removal is a classic computer vision task, but existing solutions fall into two restrictive camps:

  1. Shadow-unaware methods attempt to remove all shadows globally, often struggling to distinguish between actual shadows and naturally dark surfaces.
  2. Shadow-aware methods rely on precise, pixel-level shadow masks. These masks are tedious for humans to create and don't specify which object cast the shadow in complex scenes.

The authors identify a fundamental challenge they call object-shadow asymmetry: a user points to an object, but the region that needs editing (the shadow) is spatially separated and looks completely different in shape and texture. Traditional image editing models that rely on direct prompt-to-region alignment often fail here, either editing the object itself or failing to find the shadow at all.

Methods

The authors propose a unified framework consisting of two main components:

  • Object-Aware Generator: This stage handles the "what." It uses a frozen Qwen2.5-VL encoder to interpret multimodal prompts (visual or textual) and a diffusion-based MMDiT backbone to generate a high-quality, shadow-free "proposal" image. While this proposal looks great, because it's generated via a global diffusion process, it might introduce unintended changes to non-shadow regions.
  • Shadow-Aware Local Refiner: This stage handles the "where." It addresses the asymmetry problem by explicitly modeling the correspondence between the specified object and its cast shadow. It utilizes a SAM2 image encoder to extract multi-scale features and a "cross-scale shadow grounding" module to locate the target shadow. Finally, a context aggregation module selectively blends the generator's proposal into the original image, ensuring only the target shadow is removed.
The FreeDeshadow framework architecture, showing the two-stage process of global proposal generation and local refinement.

Datasets

To train and evaluate this new task, the authors constructed PCSRD+, a large-scale dataset that significantly advances the state of the art:

  • Scale: 154,571 quadruples (shadow image, prompt, shadow mask, and ground-truth shadow-free image).
  • Resolution: High-fidelity 1024×1024 images.
  • Prompts: Six types of prompts are supported. Visual prompts include dots, lines, and object masks. Textual prompts involve reasoning about object attributes (e.g., "the child in the green striped shirt"), spatial relations, or states.
Overview of the controllable cues supported by FreeDeshadow, ranging from sparse dots to complex textual reasoning.

Key Insights and Ideas

  • Asymmetry as a Feature, Not a Bug: By acknowledging that the prompt (object) and the edit (shadow) don't overlap, the authors created a system that "reasons" about the scene's geometry rather than just performing simple texture replacement.
  • The Power of Hybrid Architectures: Combining the generative strength of diffusion models with the precise localization of discriminative models (like SAM2) allows for both high-quality synthesis and strict spatial control.
  • Reasoning-Aware Editing: Moving from "edit this mask" to "remove the shadow of the person on the left" makes image editing much more intuitive for non-expert users.

Key Results

FreeDeshadow outperforms both traditional shadow removal methods and general-purpose image editing models (like InstructPix2Pix or ControlNet) across several metrics:

  • Quantitative: On the PCSRD+ benchmark, it achieved a PSNR of 22.018 dB in shadow regions (a ~58% error reduction compared to strong baselines) and an impressive 51.078 dB in non-shadow regions, proving it leaves the rest of the image virtually untouched.
  • Qualitative: The model successfully handles complex scenarios with multiple overlapping shadows and diverse prompt types.
  • User Preference: In a user study, FreeDeshadow was preferred 72.5% of the time over five competitive image editing methods.
Qualitative results showing the model's ability to accurately localize and remove shadows based on various prompt types.

Limitations and Future Work

While highly effective, the authors note that this work is a "concrete instance" of a broader challenge. The object-effect discrepancy isn't unique to shadows; it also applies to reflections, caustics, and localized illumination.

Future work could involve extending this framework to handle these other spatially separated visual effects. Additionally, while the model generalizes well (as shown in zero-shot tests on the WSRD dataset), extremely crowded scenes with dozens of interacting shadows remain a challenging frontier for precise object-shadow association.

Summary updated

Abstract

Existing shadow removal methods either perform global shadow removal or rely on precise shadow masks to guide restoration. Such paradigms limit their ability to selectively remove the shadow cast by a user-specified object in practical applications. To this end, we introduce an object-aware controllable shadow removal task, which allows users to specify a target object through a simple prompt and removes only its cast shadow while preserving other content. However, object-shadow asymmetry makes it challenging to establish accurate correspondence between the specified object and its spatially separated cast shadow and to confine restoration to the target shadow region. To address these challenges, we propose FreeDeshadow, a unified controllable shadow removal framework that supports both visual and textual prompts. It consists of an object-aware generator and a shadow-aware local refiner. The former interprets the user prompt and produces a high-quality shadow-free proposal, while the latter establishes the correspondence between the object and its cast shadow and constrains the generated changes to the target shadow region. To support training and evaluation, we further construct PCSRD+, a large-scale dataset containing 154,571 quadruples at 1024$\times$1024 resolution and covering six types of user prompts. The visual prompts include a dot, a line, and an object mask, while the textual prompts describe the target object using attributes, spatial relations, or states. Extensive experiments demonstrate that FreeDeshadow accurately removes the shadow of the target object under both visual and textual prompts while effectively preserving non-target regions. It also outperforms existing methods in shadow removal performance and controllability. Project Page: https://wzlwlm13.github.io/FreeDeshadow
BibTeX
@misc{wu2026freedeshadow,
  author = {Zhiliang Wu and Jihao Gu and Kun Li and Hanyu Xuan and Hehe Fan and Meng Wang and Weisi Lin},
  title = {{FreeDeshadow: Shadow Removal for User-Specified Objects With Multimodal Prompts}},
  year = {2026},
  eprint = {2611.14608},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14608},
}
Figure