S3Net: A Single Stream Structure for Depth Guided Image Relighting

Hao-Hsiang Yang, Wei-Ting Chen, Sy-Yen Kuo

Abstract

Depth guided any-to-any image relighting aims to generate a relit image from an original source image and its corresponding depth map to match the illumination setting of a guide image and its depth map. To address this challenge, we propose S3Net, a deep learning-based neural Single Stream Structure network. S3Net is an encoder-decoder model that concatenates all images and depth maps into a single input stream. The decoder incorporates attention and enhanced modules to focus on relighting-relevant regions. Experiments on challenging benchmarks demonstrate that S3Net achieved the 3rd highest SSIM in the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge.

Introduction

Image relighting involves altering the illumination and color temperature of a source image to match a target "guide" image. Unlike standard style transfer, relighting is a geometry-aware task; it requires understanding 3D structure to accurately cast new shadows and remove existing ones. To facilitate progress in this area, the NTIRE 2021 Challenge introduced depth-guided relighting, providing depth maps alongside RGB images to assist models in understanding scene geometry.

Processing high-resolution (1024x1024) RGB-D data typically necessitates dual-stream networks, which are computationally expensive and memory-intensive. S3Net addresses this by effectively fusing RGB and depth data within a single-stream architecture, significantly reducing overhead while maintaining high performance.

Methods

The proposed S3Net (Single Stream Structure network) employs several key innovations to optimize efficiency and relighting quality:

Single-Stream Early Fusion

Instead of using separate neural network branches for RGB and depth data, S3Net concatenates the source image, source depth, guide image, and guide depth into a single 8-channel input tensor. This tensor is processed by a single modified Res2Net101 backbone. This early fusion strategy allows the model to learn complex cross-modal interactions between geometry and color without the massive computational burden of dual-stream architectures.

Attention-Driven Decoder

The decoder utilizes both Spatial Attention Mechanisms (SAM) and Channel Attention Mechanisms (CAM) to help the network prioritize regions most affected by lighting changes. It also includes an "enhanced module" that uses multi-scale average pooling to capture features across different receptive fields, ensuring both global consistency and local detail.

Wavelet SSIM Loss

To successfully manipulate shadows, the network must reconstruct sharp, high-frequency structural details. The authors combine Discrete Wavelet Transform (DWT) with Structural Similarity (SSIM) loss. By decomposing the predicted and target images into different frequency sub-bands before calculating the loss, the network is forced to optimize for finer structural details. This is used alongside Charbonnier (robust L1) and VGG-based perceptual losses.

Results and Discussion

The model was evaluated on the VIDIT (Virtual Image Dataset for Illumination Transfer) dataset, which contains 15,600 high-resolution images across 390 virtual scenes rendered under 40 different illumination settings.

Qualitative results of S3Net on the VIDIT dataset. (a) Source image, (b) Source depth, (c) Guide image, (d) Guide depth, (e) Ground truth, (f) S3Net prediction.

Quantitative Performance

In the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge, S3Net proved highly competitive, securing:

  • 3rd place in SSIM
  • 4th place in PSNR
  • 2nd place in LPIPS
  • 3rd place in Mean Perceptual Score (MPS)

Ablation studies confirmed that explicitly including depth maps improved PSNR from 18.76 to 18.84. Swapping standard SSIM loss for the proposed Wavelet SSIM loss further boosted PSNR to 19.12, yielding visibly sharper shadow boundaries.

Computational Efficiency

Despite the high resolution of the input images, the single-stream architecture is relatively fast, taking an average of 2.042 seconds to generate a relit image on a single Nvidia V100 GPU.

Conclusion

S3Net provides an efficient and effective solution for depth-guided image relighting. By utilizing early fusion in a single-stream network and a specialized wavelet-based loss, it achieves top-tier results on high-resolution benchmarks. While the model currently struggles with hallucinating details in extremely deep shadow regions where front-facing depth information is insufficient, it represents a significant step forward in geometry-aware image manipulation. Future work will explore more advanced backbones to further improve feature fusion in these challenging areas.

Summary updated

Abstract

Depth guided any-to-any image relighting aims to generate a relit image from the original image and corresponding depth maps to match the illumination setting of the given guided image and its depth map. To the best of our knowledge, this task is a new challenge that has not been addressed in the previous literature. To address this issue, we propose a deep learning-based neural Single Stream Structure network called S3Net for depth guided image relighting. This network is an encoder-decoder model. We concatenate all images and corresponding depth maps as the input and feed them into the model. The decoder part contains the attention module and the enhanced module to focus on the relighting-related regions in the guided images. Experiments performed on challenging benchmark show that the proposed model achieves the 3 rd highest SSIM in the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge.
BibTeX
@inproceedings{yang2021s,
  author = {Hao-Hsiang Yang and Wei-Ting Chen and Sy-Yen Kuo},
  title = {{S3Net: A Single Stream Structure for Depth Guided Image Relighting}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=PmhXqTWhPkC},
}
Figure