Abstract
Single Image Reflection Removal (SIRR) is a fundamental challenge in computer vision, as reflections can significantly degrade image quality and hinder downstream tasks like object detection and segmentation. This paper introduces a novel training methodology centered on a multi-step loss mechanism and a specialized synthetic dataset, RefGAN. Rather than relying solely on complex architectural innovations, our approach focuses on iterative refinement and the integration of Ranged Depth Maps (RDM) as auxiliary features. By leveraging the fact that depth estimation models typically ignore reflections, our method effectively distinguishes between background layers and artifacts. Experimental results on the $SIR^2$ benchmark and real-world datasets demonstrate that our approach achieves state-of-the-art performance in both quantitative metrics and perceptual quality.
Introduction
When capturing images through glass, the resulting photograph is often a composite of two distinct layers: the background scene (transmission) and the reflected environment. Removing these reflections from a single image is a classic "ill-posed" problem because a single pixel value can be the sum of multiple light sources, making it mathematically difficult to separate the layers without additional information.
Existing solutions often struggle with "ghosting" effects or fail to handle intense, non-linear reflections. Furthermore, the field suffers from a lack of high-quality training data; capturing perfectly aligned pairs of reflected and reflection-free images in the real world is labor-intensive and technically challenging. This paper addresses these hurdles through a combination of iterative loss functions, depth-aware priors, and GAN-based data synthesis.
Methodology
Multi-step Loss Mechanism
Traditional models typically process an image in a single pass. We propose a multi-step approach where the model’s output is iteratively fed back into the network as input for subsequent steps. The loss is accumulated across these iterations. This mechanism forces the model to refine its predictions progressively. If a reflection persists after the first step, the multi-step loss effectively "amplifies" the penalty, pushing the network to be more aggressive and precise in its removal strategy. Our findings indicate that this iterative refinement is significantly more effective than simply increasing the weight of a single-step loss.
Ranged Depth Maps (RDM)
We utilize the observation that standard depth estimation models, such as MiDaS, generally fail to perceive reflections, calculating depth only for physical objects behind the glass. To leverage this, we introduce Ranged Depth Maps. By binning continuous depth values into discrete ranges (with $k=4$ identified as optimal), we create a spatial prior that helps the model identify which pixels belong to the same physical structure. This acts as a mask, guiding the network to preserve the background while isolating reflection artifacts.
Dual-Network Architecture
The system is composed of two primary modules:
- R-CNN (Reflection-CNN): Dedicated to predicting the reflection layer itself.
- T-CNN (Target-CNN): Reconstructs the clean target image by utilizing the predicted reflection from the R-CNN, the original input image, and the RDM.
Data Synthesis with RefGAN
To overcome the scarcity of realistic training data, we developed RefGAN. Traditional synthetic datasets often use linear synthesis—simply adding two images together—which fails to capture the complex physics of light. We employed a Pix2Pix GAN to generate 7,115 image pairs that simulate non-linear reflection patterns, including "foggy" and "sharpened" effects. This high-quality synthetic data allows the model to learn patterns that are much closer to real-world conditions.
Experimental Results
Performance Benchmarks
The proposed method was evaluated against state-of-the-art (SOTA) models like IBCLN and VIRR using the $SIR^2$ benchmark and the Nature dataset. Our model demonstrated superior performance, particularly in the LPIPS metric, which measures perceptual similarity to the human eye.
- Real-world Accuracy: On a test set of 50 real-world images, our model achieved a PSNR of 25.18 and an SSIM of 0.904, outperforming the previous best-in-class model (IBCLN, 24.31 PSNR).
- Perceptual Quality: The model showed marked improvements in the "Object" and "Wild" categories of the $SIR^2$ benchmark, producing cleaner images with fewer residual artifacts.
Computational Efficiency
Despite the iterative nature of the multi-step loss, the model remains highly efficient. A $336 \times 336$ resolution image can be processed in approximately 66.8 ms on a Tesla T4 GPU, making the approach viable for near-real-time image restoration applications.
Conclusion
By shifting the focus from architectural complexity to training strategy and data quality, we have developed a robust solution for single image reflection removal. The combination of multi-step loss, Ranged Depth Maps, and GAN-generated synthetic data provides a powerful framework for restoring images obscured by glass reflections. Future work will explore the integration of more complex physical light models to handle curved or tinted glass surfaces.
Summary updated