Abstract Reflection removal from photographs is an important task in computational photography and computer vision tasks involving imaging through windows. Traditionally, the problem is approached as a single reflection removal problem under controlled scenarios. This paper generalizes reflection removal to real-world scenarios with complex light interactions. We propose an efficient learning framework for supervised image reflection separation using a polarization-guided ray-tracing model and a novel loss function design. By utilizing a polarization sensor that captures four linearly polarized photos simultaneously, and combining it with a polarization-guided image formation model, we achieve unprecedented quality in image reconstruction. We demonstrate our method through extensive experiments on both real and synthetic data.
Introduction
Capturing clear photographs through glass surfaces is a persistent challenge in computer vision. The resulting images are typically a superposition of the scene behind the glass (transmission) and the scene reflected on the glass. This "unmixing" problem is inherently ill-posed. Traditional methods often assume glass is infinitely thin, but real-world glass has thickness that causes light to bounce between front and back surfaces, leading to "ghosting" artifacts—faint, shifted secondary reflections that degrade image quality.
Methodology
The proposed framework addresses these challenges by leveraging polarization and physical modeling. Light's polarization state changes predictably upon reflection or transmission, governed by Fresnel equations.
Key components of the method include:
- Polarization-Guided Image Formation: Instead of a standard sensor, the system uses a polarization sensor to capture four linear polarization angles ($0^\circ, 45^\circ, 90^\circ, 135^\circ$) simultaneously.
- Multi-bounce Ray Tracing: The model explicitly accounts for light paths reflecting and transmitting through both the front and back interfaces of the glass. This allows the system to accurately model the spatial shifts that cause ghosting.
- Neural Network Architecture: A two-stage pipeline consisting of PolarNet, which independently decomposes each polarized image, and FusionNet, which combines these results into a refined final output.
- Differentiable Polarization Simulation Engine (PSE): This is the core innovation. A physical simulator is integrated into the training loop. The network predicts the transmitted and reflected scenes, and the PSE "re-renders" what the polarized sensor should see based on those predictions. The difference between this simulation and the actual captured image becomes a "PSE Loss" that forces the network to remain physically consistent.
Datasets and Experiments
The researchers utilized two primary data sources to train and validate their model:
- Synthetic Data: 10,000 image pairs generated using the PSE. These clean images were used to simulate reflections with varying glass thicknesses, incident angles, and refractive indices.
- Real-World Dataset: 26 unique scenes captured using a Lucid Vision Labs PHX050S-Q color polarization camera. For each scene, ground truth images (without glass) and multiple blended images with glass at different angles were captured. This high-resolution dataset has been made public to support further research.
Results and Discussion
The integration of physics as a regularizer allows the model to obey optical laws rather than relying solely on data-driven patterns. This significantly reduces artifacts compared to purely neural methods. Furthermore, the four-angle polarization sensor provides the necessary constraints to resolve the ambiguity of the unmixing problem more accurately than standard RGB cameras.
Quantitatively, the method achieved a PSNR of 31.08 and SSIM of 0.911 on challenging real-world scenes, vastly outperforming previous state-of-the-art polarization methods like ReflectNet (PSNR 14.16). Qualitatively, the results show a dramatic reduction in ghosting and superior color accuracy, even in high-dynamic-range (HDR) scenes or scenes with polarized light sources like LCD screens.
Conclusion
This work demonstrates that modeling the physical properties of light—specifically multi-bounce polarization—is essential for solving complex reflection removal in real-world settings. While the current model requires some parameter estimation (like glass thickness), it represents a major leap forward in reconstruction quality. Future work will focus on automating parameter estimation for curved surfaces and optimizing the architecture for real-time performance on mobile devices.
Summary updated