TL;DR
FreeDeshadow introduces a new paradigm for image editing: object-aware controllable shadow removal. Instead of requiring precise shadow masks or removing all shadows globally, it allows users to specify a target object via multimodal prompts (like a simple dot, a line, or a descriptive sentence) and removes only that object’s cast shadow while keeping the rest of the scene intact.
What Problem Does This Paper Address?
Shadow removal is a classic computer vision task, but existing solutions fall into two restrictive camps:
- Shadow-unaware methods attempt to remove all shadows globally, often struggling to distinguish between actual shadows and naturally dark surfaces.
- Shadow-aware methods rely on precise, pixel-level shadow masks. These masks are tedious for humans to create and don't specify which object cast the shadow in complex scenes.
The authors identify a fundamental challenge they call object-shadow asymmetry: a user points to an object, but the region that needs editing (the shadow) is spatially separated and looks completely different in shape and texture. Traditional image editing models that rely on direct prompt-to-region alignment often fail here, either editing the object itself or failing to find the shadow at all.
Methods
The authors propose a unified framework consisting of two main components:
- Object-Aware Generator: This stage handles the "what." It uses a frozen Qwen2.5-VL encoder to interpret multimodal prompts (visual or textual) and a diffusion-based MMDiT backbone to generate a high-quality, shadow-free "proposal" image. While this proposal looks great, because it's generated via a global diffusion process, it might introduce unintended changes to non-shadow regions.
- Shadow-Aware Local Refiner: This stage handles the "where." It addresses the asymmetry problem by explicitly modeling the correspondence between the specified object and its cast shadow. It utilizes a SAM2 image encoder to extract multi-scale features and a "cross-scale shadow grounding" module to locate the target shadow. Finally, a context aggregation module selectively blends the generator's proposal into the original image, ensuring only the target shadow is removed.
Datasets
To train and evaluate this new task, the authors constructed PCSRD+, a large-scale dataset that significantly advances the state of the art:
- Scale: 154,571 quadruples (shadow image, prompt, shadow mask, and ground-truth shadow-free image).
- Resolution: High-fidelity 1024×1024 images.
- Prompts: Six types of prompts are supported. Visual prompts include dots, lines, and object masks. Textual prompts involve reasoning about object attributes (e.g., "the child in the green striped shirt"), spatial relations, or states.
Key Insights and Ideas
- Asymmetry as a Feature, Not a Bug: By acknowledging that the prompt (object) and the edit (shadow) don't overlap, the authors created a system that "reasons" about the scene's geometry rather than just performing simple texture replacement.
- The Power of Hybrid Architectures: Combining the generative strength of diffusion models with the precise localization of discriminative models (like SAM2) allows for both high-quality synthesis and strict spatial control.
- Reasoning-Aware Editing: Moving from "edit this mask" to "remove the shadow of the person on the left" makes image editing much more intuitive for non-expert users.
Key Results
FreeDeshadow outperforms both traditional shadow removal methods and general-purpose image editing models (like InstructPix2Pix or ControlNet) across several metrics:
- Quantitative: On the PCSRD+ benchmark, it achieved a PSNR of 22.018 dB in shadow regions (a ~58% error reduction compared to strong baselines) and an impressive 51.078 dB in non-shadow regions, proving it leaves the rest of the image virtually untouched.
- Qualitative: The model successfully handles complex scenarios with multiple overlapping shadows and diverse prompt types.
- User Preference: In a user study, FreeDeshadow was preferred 72.5% of the time over five competitive image editing methods.
Limitations and Future Work
While highly effective, the authors note that this work is a "concrete instance" of a broader challenge. The object-effect discrepancy isn't unique to shadows; it also applies to reflections, caustics, and localized illumination.
Future work could involve extending this framework to handle these other spatially separated visual effects. Additionally, while the model generalizes well (as shown in zero-shot tests on the WSRD dataset), extremely crowded scenes with dozens of interacting shadows remain a challenging frontier for precise object-shadow association.
Summary updated