Single-image reflection removal via self-supervised diffusion models

Zhengyang Lu, Weifan Wang, Tianhao Guo, Feng Wang

TL;DR

This paper introduces a self-supervised framework for removing reflections from single images by combining the generative power of Denoising Diffusion Probabilistic Models (DDPMs) with cycle-consistency. By leveraging a dual-diffusion decomposition process, the method effectively separates reflections from background scenes without requiring paired real-world training data, achieving state-of-the-art results on complex, real-world datasets.

What Problem Does This Paper Address?

Single Image Reflection Removal (SIRR) is a classic "ill-posed" problem in computer vision: a single captured image is the sum of a background (transmission) layer and a reflected layer, and there are infinite ways to split them.

Existing solutions generally fall into two camps:

  1. Model-based methods: These rely on hand-crafted mathematical priors (like assuming reflections are blurrier than the background). They often fail in the "wild" where reflections are sharp or complex.
  2. Data-driven methods: These use deep learning but typically require massive amounts of paired data (the same scene with and without reflection). Since capturing such pairs in the real world is incredibly difficult, models trained on synthetic data often fail to generalize to real-world physics.

The authors aim to bridge this gap by creating a model that can learn from unpaired real-world images using self-supervision.

Methods

The proposed architecture consists of three main pillars:

  • Reflective Removal Network (RRN): This is the core "separator." It utilizes the DDPM framework to model the decomposition. Unlike standard diffusion that turns noise into an image, this RRN learns to iteratively "denoise" the input image into two distinct components: the transmission image ($s$) and the reflection image ($r$).
  • Reflective Synthesis Network (RSN): To enable self-supervision, the model must be able to put the pieces back together. The RSN uses a nonlinear attention-based mechanism to re-synthesize the original input from the separated $s$ and $r$ layers. This ensures that no information is lost during the removal process.
  • Transmission Discriminator (TD): To prevent the model from "cheating" during unpaired training (e.g., just passing the input through unchanged), an adversarial discriminator is used to ensure the recovered transmission image looks like a real, reflection-free photograph.

The training follows a cycle-consistency logic: an image is decomposed, then re-synthesized, and the model is penalized if the re-synthesized version doesn't match the original input.

Datasets

The authors evaluated their model on several benchmarks and introduced a significant new one:

  • SIR²: A standard dataset of 500 real-world image pairs.
  • FRR (Flash-Based Reflection Removal): Real-world scenes captured with and without flash.
  • Museum Reflection Removal (MRR) [New]: A newly introduced dataset specifically targeting museum environments (glass cases, framed art). It contains 1,621 high-quality unpaired images and 721 paired images, providing a challenging benchmark for heritage preservation and digital archiving.

Key Insights and Ideas

  • Diffusion as a Separator: While diffusion models are usually seen as generators, this paper treats the reflection removal task as a "decomposition" problem within the diffusion latent space. This allows the model to handle the multimodal nature of reflections (where one input could have multiple valid outputs).
  • The RAM Metric: The authors argue that standard metrics like PSNR and SSIM don't always capture residual "ghosting" artifacts. They propose the Reflection Artifact Measure (RAM), which uses Fourier Transform differences to specifically quantify high-frequency reflection remnants in the recovered image.
  • Attention-Based Fusion: Instead of a simple linear addition ($I = T + R$), the RSN uses spatial attention maps to adaptively fuse layers. This acknowledges that reflections vary in intensity across the frame, allowing for more realistic reconstruction.

Key Results

The proposed method consistently outperformed state-of-the-art (SOTA) techniques like CEILNet, BDN, and SDN across all datasets:

  • Quantitative Gains: On the SIR² dataset, the model achieved a PSNR of 28.41 dB, a significant improvement over the previous SOTA of 27.91 dB.
  • Artifact Suppression: The model showed a marked decrease in the new RAM metric (0.0549 vs. 0.0599 for the next best), indicating it is much better at removing subtle "ghost" reflections that other models leave behind.
  • Qualitative Superiority: Visual results (especially in the MRR dataset) show the model successfully recovering intricate textures in fossils and paintings that were previously obscured by harsh glass reflections.

Limitations and Future Work

Despite the strong results, the authors note a few hurdles:

  • Low-Contrast Scenarios: When reflections are very faint or their patterns perfectly align with the background texture, the model can struggle to distinguish between the two.
  • Computational Overhead: With 89.7M parameters, the model is slightly heavier than some baselines. While it runs efficiently on modern GPUs (0.235s for a 512x512 image), it may not be ready for real-time mobile applications yet.
  • Future Directions: The authors suggest incorporating physical reflection models (like Fresnel equations) and multi-scale attention to further improve performance on high-resolution images.

Summary updated

Abstract

Reflections often degrade the visual quality of images captured through transparent surfaces, and reflection removal methods suffers from the shortage of paired real-world samples.This paper proposes a hybrid approach that combines cycle-consistency with denoising diffusion probabilistic models (DDPM) to effectively remove reflections from single images without requiring paired training data. The method introduces a Reflective Removal Network (RRN) that leverages DDPMs to model the decomposition process and recover the transmission image, and a Reflective Synthesis Network (RSN) that re-synthesizes the input image using the separated components through a nonlinear attention-based mechanism. Experimental results demonstrate the effectiveness of the proposed method on the SIR$^2$, Flash-Based Reflection Removal (FRR) Dataset, and a newly introduced Museum Reflection Removal (MRR) dataset, showing superior performance compared to state-of-the-art methods.
BibTeX
@article{lu2024single,
  author = {Zhengyang Lu and Weifan Wang and Tianhao Guo and Feng Wang},
  title = {{Single-image reflection removal via self-supervised diffusion models}},
  journal = {The Journal of Supercomputing},
  year = {2024},
  doi = {10.1007/s11227-024-06837-9},
}

Loading related content...

Figure