- A Tailored Approach to Single Image Reflection Removal
Abstract: Removing reflection artefacts from a single image is a problem of both theoretical and practical interest, which still presents challenges because of the massively ill-posed nature of the problem. In this work, we propose a technique based on a novel optimisation problem. Firstly, we introduce a simple user interaction scheme, which helps minimise information loss in reflection-free regions. Secondly, we introduce an $H^2$ fidelity term, which preserves fine detail while enforcing global colour similarity. We show that this combination allows us to mitigate some major drawbacks of the existing methods for reflection removal. We demonstrate, through numerical and visual experiments, that our method is able to outperform the state-of-the-art methods and compete with recent deep-learning approaches.
TL;DR
This paper introduces a novel optimization-based approach for removing reflections from a single image by combining a sophisticated $H^2$ fidelity term with minimal user guidance. By using a "spatially aware" prior that tells the algorithm where reflections are located, the method outperforms traditional models and holds its own against—or even exceeds—the visual quality of state-of-the-art deep learning techniques.
What Problem Does This Paper Address?
Taking a photo through a glass window often results in unwanted reflections. Mathematically, this is a "massively ill-posed" problem: you have one observed image ($Y$) that is the sum of two unknown layers—the transmission ($T$, what you want) and the reflection ($R$, what you don't).
While specialized hardware (like polarizers) or multi-shot sequences can help, most users only have a single, standard image. Existing single-image methods often struggle with "flattening" (loss of texture), unnatural color shifts, or creating "ghosting" artifacts. Deep learning (DL) approaches have improved results but often lack the structural constraints needed to prevent inconsistent artifacts or "hallucinations" in the final output.
Methods
The authors propose a tailored optimization framework that relies on two key innovations:
- The $H^2$ Fidelity Term: Instead of just looking at pixel differences, the authors use a combined fidelity term. They employ a Laplacian ($\Delta$) to penalize differences in the second derivative, which helps preserve high-frequency details, and an $L^2$ norm to ensure global color similarity, preventing the "color drift" common in previous gradient-based methods.
- Spatially Aware Prior ($\phi$): The authors argue that humans are great at spotting reflections, even if they aren't good at removing them. They introduce a simple user-interaction scheme where a user "scribbles" over reflection-heavy areas. This creates a mask ($\phi$) that tells the algorithm exactly where to apply the sparsity prior, leaving reflection-free regions untouched and sharp.
The resulting problem is solved using half-quadratic splitting, an iterative technique that breaks the complex optimization into smaller, computationally tractable sub-problems solved via the ADAM optimizer.
Datasets and Results
The researchers evaluated their method using the SIR$^2$ Benchmark (1,500 real-world images), the Berkeley Dataset (110 image pairs), and various unconstrained "Real-World" photographs.
Key Insights
- Efficiency: The user-interaction component was shown to be 700% faster than previous interactive benchmarks, taking only ~5 seconds per image while producing superior numerical metrics.
- Mathematical Rigor: The model-based approach provides more stable, "natural-looking" results compared to Deep Learning (DL) models. In tests against DL models like FAN17 and ZHANG18, the proposed method preserved fine details—like the texture of a shirt collar—that neural networks often blurred.
- Quantitative Performance: On the SIR$^2$ dataset, the method consistently achieved higher scores in SSIM (Structural Similarity) and PSNR (Peak Signal-to-Noise Ratio) compared to other model-based methods like AR17 and LB14.
Limitations and Future Work
The primary limitation is the reliance on user input. While the input is "crude" and fast, it still prevents the method from being a fully automated solution. The authors suggest that future work could involve using a Deep Learning pre-processor to automatically generate the region selection mask ($\phi$), combining automated detection with high-quality optimization-based reconstruction.
Summary updated