S3Net: A Single Stream Structure for Depth Guided Image Relighting

Hao-Hsiang Yang, Wei-Ting Chen, Sy-Yen Kuo

Abstract

Depth guided any-to-any image relighting aims to generate a relit image from the original image and corresponding depth maps to match the illumination setting of the given guided image and its depth map. To the best of our knowledge, this task is a new challenge that has not been addressed in the previous literature. To address this issue, we propose a deep learning-based neural Single Stream Structure network called S3Net for depth guided image relighting. This network is an encoder-decoder model. We concatenate all images and corresponding depth maps as the input and feed them into the model. The decoder part contains the attention module and the enhanced module to focus on the relighting-related regions in the guided images. Experiments performed on challenging benchmark show that the proposed model achieves the 3rd highest SSIM in the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge.

Introduction

Image relighting involves altering the illumination of a scene—such as light source position, color temperature, and shadow distribution—while preserving the underlying geometry and textures. This paper addresses the depth-guided any-to-any relighting task. Unlike simpler "one-to-one" relighting, "any-to-any" requires a model to extract the lighting "style" from an arbitrary guide image and apply it to a source image.

This task is particularly challenging as it necessitates:

  1. Shadow Removal: Eliminating existing shadows from the source image.
  2. Shadow Synthesis: Generating new shadows consistent with the 3D geometry provided by depth maps.
  3. Color Matching: Accurately replicating the color temperature of the guide image.
Overview of the depth-guided any-to-any relighting task: (a) source image, (b) source depth, (c) guide image, (d) guide depth, (e) ground truth, and (f) S3Net result.

Previous approaches often utilized dual-stream backbones to process RGB and depth data separately. However, such architectures are computationally intensive, especially for high-resolution (1024x1024) images.

Methodology: S3Net

The proposed S3Net is an encoder-decoder architecture optimized for efficiency and multi-scale feature awareness.

Single Stream Backbone

To minimize computational overhead, S3Net employs a single-stream design. The source RGB+Depth and the guide RGB+Depth are concatenated into a single 8-channel tensor. This input is processed by a Res2Net101 encoder, which excels at representing multi-scale features at a granular level. This early fusion allows the network to learn correlations between geometry (depth) and appearance (RGB) from the initial layers.

Attention-Driven Decoder

The decoder incorporates spatial and channel attention modules to direct the network's focus toward regions critical for relighting, such as areas where new shadows should be cast. Additionally, an Enhanced Module utilizes average pooling at multiple strides to capture features across various scales, ensuring the model handles both global ambient light and local highlights effectively.

Wavelet-Based Loss Function

A key technical innovation is the use of a Wavelet SSIM loss. By applying the Discrete Wavelet Transform (DWT), the model decomposes images into frequency sub-bands. This allows the loss function to penalize errors in high-frequency details, such as sharp shadow edges, more effectively than standard pixel-wise or spatial SSIM losses.

Experiments and Results

The model was evaluated using the Virtual Image Dataset for Illumination Transfer (VIDIT), which contains 390 virtual scenes, each with 40 different illumination settings (5 color temperatures and 8 azimuthal light angles).

Performance in NTIRE 2021 Challenge

S3Net was a top-tier performer in the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge:

  • Rankings: Secured 3rd place in both SSIM (Structural Similarity) and MPS (Mean Perceptual Score).
  • Quantitative Metrics: On the VIDIT dataset, the model achieved a PSNR of 19.1281 and an SSIM of 0.6969.
  • Efficiency: The model processes a 1024x1024 image in approximately 2.042 seconds, demonstrating practical utility for high-resolution applications.

Discussion and Insights

Ablation studies confirmed that depth maps are a "force multiplier," improving SSIM from 0.6821 to 0.6913 by providing the geometric "skeleton" necessary for shadow placement. Furthermore, applying SSIM in the Wavelet domain helped reconstruct fine-grained structural details often lost in low-level vision tasks.

Limitations

A notable failure case occurs in regions of large, dark shadows in the source image. If the source contains pitch-black areas, the model struggles to recover textures hidden within them. Consequently, relighting these regions often results in poor structural reconstruction due to the lack of input information. Future work will focus on more robust backbones to further improve the fusion of image and depth features.

Summary updated

Abstract

Depth guided any-to-any image relighting aims to generate a relit image from the original image and corresponding depth maps to match the illumination setting of the given guided image and its depth map. To the best of our knowledge, this task is a new challenge that has not been addressed in the previous literature. To address this issue, we propose a deep learning-based neural Single Stream Structure network called S3Net for depth guided image relighting. This network is an encoder-decoder model. We concatenate all images and corresponding depth maps as the input and feed them into the model. The decoder part contains the attention module and the enhanced module to focus on the relighting-related regions in the guided images. Experiments performed on challenging benchmark show that the proposed model achieves the 3^ rd highest SSIM in the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge.
BibTeX
@inproceedings{yang2021s,
  author = {Hao-Hsiang Yang and Wei-Ting Chen and Sy-Yen Kuo},
  title = {{S3Net: A Single Stream Structure for Depth Guided Image Relighting}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=PmhXqTWhPkC},
}
Figure