Abstract
Depth guided any-to-any image relighting aims to generate a relit image from an original source image and its corresponding depth map to match the illumination setting of a guide image and its depth map. To address this challenge, we propose S3Net, a deep learning-based neural Single Stream Structure network. S3Net is an encoder-decoder model that concatenates all images and depth maps into a single input stream. The decoder incorporates attention and enhanced modules to focus on relighting-relevant regions. Experiments on challenging benchmarks demonstrate that S3Net achieved the 3rd highest SSIM in the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge.
Introduction
Image relighting involves altering the illumination and color temperature of a source image to match a target "guide" image. Unlike standard style transfer, relighting is a geometry-aware task; it requires understanding 3D structure to accurately cast new shadows and remove existing ones. To facilitate progress in this area, the NTIRE 2021 Challenge introduced depth-guided relighting, providing depth maps alongside RGB images to assist models in understanding scene geometry.
Processing high-resolution (1024x1024) RGB-D data typically necessitates dual-stream networks, which are computationally expensive and memory-intensive. S3Net addresses this by effectively fusing RGB and depth data within a single-stream architecture, significantly reducing overhead while maintaining high performance.
Methods
The proposed S3Net (Single Stream Structure network) employs several key innovations to optimize efficiency and relighting quality:
Single-Stream Early Fusion
Instead of using separate neural network branches for RGB and depth data, S3Net concatenates the source image, source depth, guide image, and guide depth into a single 8-channel input tensor. This tensor is processed by a single modified Res2Net101 backbone. This early fusion strategy allows the model to learn complex cross-modal interactions between geometry and color without the massive computational burden of dual-stream architectures.
Attention-Driven Decoder
The decoder utilizes both Spatial Attention Mechanisms (SAM) and Channel Attention Mechanisms (CAM) to help the network prioritize regions most affected by lighting changes. It also includes an "enhanced module" that uses multi-scale average pooling to capture features across different receptive fields, ensuring both global consistency and local detail.
Wavelet SSIM Loss
To successfully manipulate shadows, the network must reconstruct sharp, high-frequency structural details. The authors combine Discrete Wavelet Transform (DWT) with Structural Similarity (SSIM) loss. By decomposing the predicted and target images into different frequency sub-bands before calculating the loss, the network is forced to optimize for finer structural details. This is used alongside Charbonnier (robust L1) and VGG-based perceptual losses.
Results and Discussion
The model was evaluated on the VIDIT (Virtual Image Dataset for Illumination Transfer) dataset, which contains 15,600 high-resolution images across 390 virtual scenes rendered under 40 different illumination settings.
Quantitative Performance
In the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge, S3Net proved highly competitive, securing:
- 3rd place in SSIM
- 4th place in PSNR
- 2nd place in LPIPS
- 3rd place in Mean Perceptual Score (MPS)
Ablation studies confirmed that explicitly including depth maps improved PSNR from 18.76 to 18.84. Swapping standard SSIM loss for the proposed Wavelet SSIM loss further boosted PSNR to 19.12, yielding visibly sharper shadow boundaries.
Computational Efficiency
Despite the high resolution of the input images, the single-stream architecture is relatively fast, taking an average of 2.042 seconds to generate a relit image on a single Nvidia V100 GPU.
Conclusion
S3Net provides an efficient and effective solution for depth-guided image relighting. By utilizing early fusion in a single-stream network and a specialized wavelet-based loss, it achieves top-tier results on high-resolution benchmarks. While the model currently struggles with hallucinating details in extremely deep shadow regions where front-facing depth information is insufficient, it represents a significant step forward in geometry-aware image manipulation. Future work will explore more advanced backbones to further improve feature fusion in these challenging areas.
Summary updated