Reflection Separation Using Guided Annotation

Ofer Springer, Yair Weiss

Abstract

Photographs taken through a glass surface often contain an approximately linear superposition of reflected and transmitted layers. Decomposing an image into these layers is generally an ill-posed task and the use of an additional image prior and user provided cues is presently necessary in order to obtain good results. Current annotation approaches rely on a strong sparsity assumption. For images with significant texture this assumption does not typically hold, thus rendering the annotation process unviable. In this paper we show that using a Gaussian Mixture Model patch prior, the correct local decomposition can almost always be found as one of 100 likely modes of the posterior. Thus, the user need only choose one of these modes in a sparse set of patches and the decomposition may then be completed automatically. We demonstrate the performance of our method using synthesized and real reflection images.

Introduction: The Challenge of Reflection

When capturing a scene through a glass pane, the resulting image $y$ is typically a linear combination of the transmitted scene $x_1$ and the reflected scene $x_2$: $y = x_1 + x_2$. This presents a classic ill-posed problem in computer vision: for every pixel, we have one known value but two unknown components.

Previous user-assisted solutions, such as the method by Levin and Weiss (LW), relied heavily on a sparsity assumption. This assumption posits that at any given edge, the detail belongs to only one of the two layers. Users were required to manually label edges to guide the separation. However, this approach fails in the presence of significant texture, where edges from both layers overlap and the sparsity assumption no longer holds.

Analysis of gradient filter responses for textured images, showing that the sparsity assumption often fails in natural scenes.

Methodology: GMM-C and Guided Annotation

The authors propose a new framework, GMM-C (Gaussian Mixture Model with component annotations), which moves away from edge labeling toward selecting "modes" of a probability distribution.

The GMM Patch Prior

The system utilizes a Gaussian Mixture Model trained on $8 \times 8$ patches from the BSDS300 dataset. This prior captures the statistical structure of natural image patches, allowing the model to understand what "clean" textures look like.

Guided Annotation via Posterior Modes

Given a mixed input patch $y$, the authors derive the posterior probability $Pr(x_1|y)$. A key discovery is that while the single most likely decomposition (MAP estimate) often fails, the correct local decomposition is almost always present within the top 100 modes of this posterior.

In the guided annotation phase, the system presents the user with these top 100 candidate decompositions for a few selected patches. The user simply identifies the most accurate "unmixed" version. These "anchor" patches then guide a global optimization process using the Expected Patch Log Likelihood (EPLL) framework to reconstruct the full layers.

Key Results and Evaluation

The GMM-C method was evaluated against the LW baseline using both synthetic and real-world data.

  • Quantitative Performance: In synthetic benchmarks involving 8,000 patch pairs, GMM-C outperformed the LW method by 3.3 to 4.6 dB in PSNR.
  • Qualitative Superiority: On real images with complex textures (e.g., reflections on patterned clothing), the GMM-C method successfully separated layers without the "texture leak" common in sparsity-based methods.
Comparison between the proposed GMM-C method and the baseline LW method on a synthetic reflection image.

An ablation study revealed that the performance boost is dual-sourced: approximately 2.5 dB is attributed to the superior GMM prior, while 1.6 dB stems from the more effective guided annotation interface.

Conclusion and Future Directions

By leveraging a GMM patch prior and a human-in-the-loop mode selection process, the GMM-C method effectively addresses the limitations of sparsity-based reflection separation. While the process currently requires user interaction and takes roughly two minutes per image, it offers a robust solution for highly textured scenes. Future work may explore multi-scale approaches to handle larger structures or apply this "mode selection" paradigm to other restoration tasks like image deblurring or denoising.

Summary updated

Abstract

Photographs taken through a glass surface often contain an approximately linear superposition of reflected and transmitted layers. Decomposing an image into these layers is generally an ill-posed task and the use of an additional image prior and user provided cues is presently necessary in order to obtain good results. Current annotation approaches rely on a strong sparsity assumption. For images with significant texture this assumption does not typically hold, thus rendering the annotation process unviable. In this paper we show that using a Gaussian Mixture Model patch prior, the correct local decomposition can almost always be found as one of 100 likely modes of the posterior. Thus, the user need only choose one of these modes in a sparse set of patches and the decomposition may then be completed automatically. We demonstrate the performance of our method using synthesized and real reflection images.
BibTeX
@inproceedings{springer2017reflection,
  author = {Ofer Springer and Yair Weiss},
  title = {{Reflection Separation Using Guided Annotation}},
  booktitle = {ICIP},
  year = {2017},
  url = {https://openreview.net/forum?id=DoECP_gWiR3},
}
Figure