RenDetNet: Weakly-Supervised Shadow Detection with Shadow Caster Verification

Nikolina Kubiak, Elliot Wortman, Armin Mustafa, Graeme Phillipson, Stephen Jolly, Simon Hadfield

Abstract

Shadow detection is a critical yet challenging task in computer vision, primarily due to the inherent ambiguity between dark-colored textures and actual shadows. Traditional supervised methods are limited by the scarcity and noise of human-annotated datasets. This paper introduces RenDetNet, a novel weakly-supervised framework that leverages physics-based verification. By identifying the "caster" (the object creating the shadow) and using a differentiable renderer to simulate its removal, the model learns to detect shadows without manual labels. Experimental results on synthetic datasets demonstrate that RenDetNet achieves a superior balance of precision and recall compared to existing state-of-the-art models.

Introduction

The primary hurdle in shadow detection is the "dark vs. shadow" ambiguity. Standard convolutional neural networks often misidentify black objects or dark textures as shadows because they lack an understanding of the underlying scene geometry and lighting conditions. Furthermore, the reliance on fully supervised learning has created a bottleneck; manual annotation is labor-intensive and often inconsistent across different human labelers.

Shadow detection comparison on a complex mosaic floor, highlighting the challenge of distinguishing dark textures from true shadows.

RenDetNet proposes a shift toward self-supervised, physics-backed systems. By asking whether a predicted shadow has a corresponding physical cause (a caster), the model can verify its own predictions during training.

Methodology

RenDetNet utilizes a fully-convolutional architecture with a shared encoder and two specialized decoder heads: one for the shadow mask (SM) and one for the caster mask (CM).

Shadow Caster Verification

The defining feature of the training process is the Shadow Caster Verification pipeline. When the network predicts an SM and CM from a rendered image, the system performs the following steps:

  1. Mesh Carving: The predicted CM is used to identify and "carve out" the corresponding parts of the 3D scene mesh.
  2. Differentiable Re-rendering: The scene is re-rendered without these carved-out objects.
  3. Verification: The difference between the original and the "carved" render ($\Delta I$) provides a ground-truth signal. If the network's predicted masks align with this physical change, the prediction is considered verified.

Self-Supervised Losses

To further refine the masks, the authors employ additional losses:

  • Caster Loss ($L_{cm}$): Ensures the caster mask remains within object boundaries by using reflectance-only renders.
  • Shadow Loss ($L_{sm}$): Isolates shadow regions by comparing renders with varying light directions.
  • Color Space Optimization: Training in the Lab color space was found to significantly outperform RGB, as the 'L' channel (lightness) allows the network to better isolate illumination changes from texture variations.

Datasets

Since the verification pipeline requires 3D scene information, the authors developed two synthetic datasets using the Mitsuba 3 renderer. The first features 3D virtual humans in fixed poses, while the second (which has been released publicly) utilizes BEDLAM humans and various clutter objects with diverse poses and heavily textured floors.

Key Results

The performance of RenDetNet was evaluated using the Balanced Error Rate (BER). In tests on the virtual human dataset, RenDetNet achieved a BER of 6.344, significantly outperforming FDRNet (7.276) and BDRAR (8.619).

Qualitative results for a virtual human scene, showing the input, ground truth, and RenDetNet's estimated shadow mask.

On more complex, heavily textured datasets, RenDetNet maintained a competitive BER (11.992) while demonstrating superior qualitative performance. Unlike more aggressive models that might "spill" shadow boundaries, RenDetNet finds a "happy medium," preserving fine details and maintaining high precision. Ablation studies confirmed that the staged introduction of the rendering-based verification loss ($L_{ren}$) is essential for achieving these results.

Conclusion and Future Work

RenDetNet demonstrates the power of using physics as a supervisor in computer vision tasks. While the model currently relies on synthetic data and requires 3D meshes during training, it paves the way for more robust, self-supervised shadow detection. Future research will focus on bridging the synthetic-to-real gap and exploring hybrid training regimes that combine self-supervision with limited real-world annotated data to improve generalization to complex global illumination and camera noise.

Summary updated

BibTeX
@inproceedings{kubiak2024rendetnet,
  author = {Nikolina Kubiak and Elliot Wortman and Armin Mustafa and Graeme Phillipson and Stephen Jolly and Simon Hadfield},
  title = {{RenDetNet: Weakly-Supervised Shadow Detection with Shadow Caster Verification}},
  booktitle = {Workshops @ ECCV},
  year = {2024},
  doi = {10.1007/978-3-031-91838-4\_11},
}
Figure