Single-image reflection removal via self-supervised diffusion models

Zhengyang Lu, Weifan Wang, Tianhao Guo, Feng Wang

Abstract

Reflections often degrade the visual quality of images captured through transparent surfaces, and reflection removal methods suffer from the shortage of paired real-world samples. This paper proposes a hybrid approach that combines cycle-consistency with denoising diffusion probabilistic models (DDPM) to effectively remove reflections from single images without requiring paired training data. The method introduces a Reflective Removal Network (RRN) that leverages DDPMs to model the decomposition process and recover the transmission image, and a Reflective Synthesis Network (RSN) that re-synthesizes the input image using the separated components through a nonlinear attention-based mechanism. Experimental results demonstrate the effectiveness of the proposed method on the SIR$^2$, Flash-Based Reflection Removal (FRR) Dataset, and a newly introduced Museum Reflection Removal (MRR) dataset, showing superior performance compared to state-of-the-art methods.

Introduction

Single Image Reflection Removal (SIRR) is a classic "ill-posed" problem in computer vision: a single captured image is the sum of a background (transmission) layer and a reflected layer, and there are infinite ways to split them. Existing solutions generally fall into two camps: model-based methods relying on hand-crafted priors, and data-driven methods requiring massive amounts of paired data. Since capturing such pairs in the real world is incredibly difficult, models trained on synthetic data often fail to generalize to real-world physics.

This paper introduces a self-supervised framework for removing reflections from single images by combining the generative power of Denoising Diffusion Probabilistic Models (DDPMs) with cycle-consistency. By leveraging a dual-diffusion decomposition process, the method effectively separates reflections from background scenes without requiring paired real-world training data.

Methodology

The proposed architecture consists of three main pillars: the Reflective Removal Network (RRN), the Reflective Synthesis Network (RSN), and a Transmission Discriminator (TD).

Dual-Diffusion Decomposition

The RRN is the core "separator." It utilizes the DDPM framework to model the decomposition process. Unlike standard diffusion that turns noise into an image, this RRN learns to iteratively "denoise" the input image into two distinct components: the transmission image ($s$) and the reflection image ($r$).

The dual-diffusion architecture showing the parallel Transmission and Reflection DDPM modules used for image decomposition.

Self-Supervised Cycle-Consistency

To enable self-supervision, the model must be able to put the pieces back together. The RSN uses a nonlinear attention-based mechanism to re-synthesize the original input from the separated layers. This ensures that no information is lost during the removal process. Finally, a Transmission Discriminator (TD) is used to ensure the recovered transmission image looks like a real, reflection-free photograph, preventing the model from simply passing the input through unchanged.

Datasets and Evaluation

The authors evaluated their model on several benchmarks and introduced a significant new one: the Museum Reflection Removal (MRR) dataset. Targeting museum environments (glass cases, framed art), it contains 1,621 high-quality unpaired images and 721 paired images, providing a challenging benchmark for heritage preservation.

To address the limitations of standard metrics like PSNR, the authors proposed the Reflection Artifact Measure (RAM). This metric uses Fourier Transform differences to specifically quantify high-frequency reflection remnants, better capturing residual "ghosting" artifacts that other models might leave behind.

Key Results

The proposed method consistently outperformed state-of-the-art (SOTA) techniques like CEILNet, BDN, and SDN across all datasets. On the SIR² dataset, the model achieved a PSNR of 28.41 dB, a significant improvement over the previous SOTA of 27.91 dB.

Qualitative results and examples from the Museum Reflection Removal (MRR) dataset highlighting the complexity of real-world reflections.

Visual results, especially in the MRR dataset, show the model successfully recovering intricate textures in fossils and paintings that were previously obscured by harsh glass reflections. The attention-based fusion in the RSN allows for more realistic reconstruction by acknowledging that reflections vary in intensity across the frame.

Conclusion and Future Work

This work demonstrates that treating reflection removal as a decomposition problem within the diffusion latent space is highly effective for handling real-world complexity. While the model shows qualitative and quantitative superiority, future work will address computational overhead (currently 89.7M parameters) and performance in extremely low-contrast scenarios where reflections perfectly align with background textures.

Summary updated

Abstract

Reflections often degrade the visual quality of images captured through transparent surfaces, and reflection removal methods suffers from the shortage of paired real-world samples.This paper proposes a hybrid approach that combines cycle-consistency with denoising diffusion probabilistic models (DDPM) to effectively remove reflections from single images without requiring paired training data. The method introduces a Reflective Removal Network (RRN) that leverages DDPMs to model the decomposition process and recover the transmission image, and a Reflective Synthesis Network (RSN) that re-synthesizes the input image using the separated components through a nonlinear attention-based mechanism. Experimental results demonstrate the effectiveness of the proposed method on the SIR$^2$, Flash-Based Reflection Removal (FRR) Dataset, and a newly introduced Museum Reflection Removal (MRR) dataset, showing superior performance compared to state-of-the-art methods.
BibTeX
@article{lu2024single,
  author = {Zhengyang Lu and Weifan Wang and Tianhao Guo and Feng Wang},
  title = {{Single-image reflection removal via self-supervised diffusion models}},
  journal = {The Journal of Supercomputing},
  year = {2024},
  doi = {10.1007/s11227-024-06837-9},
}
Figure