Reflection separation using guided annotation

Ofer Springer, Yair Weiss

TL;DR

Separating a reflection from a photograph (like a shop window) is a classic "ill-posed" problem because one image is actually the sum of two. This paper introduces a semi-automatic method that uses a Gaussian Mixture Model (GMM) to suggest potential ways to "unmix" small patches of the image; the user simply picks the best-looking suggestion, allowing the algorithm to accurately reconstruct the two separate layers even in highly textured scenes where previous methods failed.

What Problem Does This Paper Address?

When you take a photo through glass, the resulting image $y$ is a linear combination of the scene behind the glass ($x_1$) and the reflection on the surface ($x_2$). Mathematically, $y = x_1 + x_2$. Since we have one equation and two unknowns for every pixel, the problem is impossible to solve without extra information.

Previous "user-assisted" methods, most notably by Levin and Weiss (LW), relied on a sparsity assumption: the idea that at any given edge in the photo, the detail belongs to only one of the two layers. Users would manually label which edges belonged to which layer. However, this fails miserably when both layers are highly textured (e.g., a reflection of a tree on a patterned shirt). In these cases, edges overlap, the sparsity assumption breaks down, and the user cannot provide accurate labels.

Methods

The authors propose a new framework called GMM-C (GMM with component annotations). Their approach shifts from labeling edges to selecting "modes" of a probability distribution.

  1. The GMM Patch Prior: They utilize a Gaussian Mixture Model trained on $8 \times 8$ patches of natural images. This model understands what "normal" image patches look like.
  2. The Patch Posterior: Given a blurry or mixed input patch $y$, the authors derive a mathematical way to calculate the posterior probability $Pr(x_1|y)$. This resulting distribution is itself a GMM with a large number of components ($K^2$).
  3. Guided Annotation: Instead of asking the user to identify edges, the system extracts a patch and presents the user with the top 100 most likely "decompositions" (modes) suggested by the GMM. The user simply picks the one that looks like a clean version of one of the layers.
  4. Global Optimization: Once the user provides a few of these "anchor" patches, the algorithm propagates this information across the whole image using the Expected Patch Log Likelihood (EPLL) framework, which ensures the final separated images are globally consistent and look like natural images.

Datasets

  • BSDS300 (Berkeley Segmentation Dataset): This is the primary dataset used. The authors used the training set to learn the GMM parameters for $8 \times 8$ patches.
  • Synthetic Evaluation: They created a benchmark by randomly pairing 8,000 $40 \times 40$ patches from the BSDS300 test set to simulate reflection images.
  • Real-World Images: The paper demonstrates results on several real-world photographs containing complex reflections and textures.

Key Insights and Ideas

  • The "Top 100" Rule: The most striking insight is that while the "most likely" single decomposition (the MAP estimate) often fails, the correct local decomposition is almost always hidden within the top 100 modes of the GMM posterior.
  • Human-in-the-loop Efficiency: It is much easier for a human to look at a few small candidate patches and say "that one looks right" than to manually trace edges or guess which layer a specific pixel belongs to.
  • Texture vs. Sparsity: By using a GMM, the model captures the statistics of texture much better than simple derivative filters. This allows the system to handle "non-sparse" regions like asphalt, tree bark, or fur, which were previously impossible to separate.

Key Results

  • Quantitative Gain: In synthetic tests, the GMM-C method outperformed the previous state-of-the-art (LW method) by 3.3 to 4.6 dB in terms of Peak Signal-to-Noise Ratio (PSNR).
  • Ablation Study: The authors found that the improvement came from two places: about 2.5 dB was due to the superior GMM prior itself, and 1.6 dB was due to the more effective "guided annotation" interface.
  • Qualitative Success: On real images, the method successfully separated reflections from highly textured foregrounds (like a mannequin's shirt) where the sparsity-based LW method caused "texture leak," where details from one layer remained visible in the other.

Limitations and Future Work

  • User Interaction Required: While the annotation is "guided," it is not fully automatic. A human must still select the points and the correct modes.
  • Computational Speed: The optimization process takes about two minutes for a standard image using non-optimized Matlab code. While acceptable for professional editing, it isn't yet "instant."
  • Patch Size Constraints: The model relies on $8 \times 8$ patches. While effective for local texture, very large-scale reflections or structures might benefit from a multi-scale approach or larger patch sizes.
  • Future Direction: The authors suggest that this "mode selection" annotation style could potentially be applied to other ill-posed image restoration tasks beyond just reflection separation.

Summary updated

Abstract

Photographs taken through a glass surface often contain an approximately linear superposition of reflected and transmitted layers. Decomposing an image into these layers is generally an ill-posed task and the use of an additional image prior and user provided cues is presently necessary in order to obtain good results. Current annotation approaches rely on a strong sparsity assumption. For images with significant texture this assumption does not typically hold, thus rendering the annotation process unviable. In this paper we show that using a Gaussian Mixture Model patch prior, the correct local decomposition can almost always be found as one of 100 likely modes of the posterior. Thus, the user need only choose one of these modes in a sparse set of patches and the decomposition may then be completed automatically. We demonstrate the performance of our method using synthesized and real reflection images.
BibTeX
@inproceedings{springer2017reflection,
  author = {Ofer Springer and Yair Weiss},
  title = {{Reflection separation using guided annotation}},
  booktitle = {ICIP},
  year = {2017},
  url = {https://openreview.net/forum?id=DoECP_gWiR3},
}

Loading related content...

Figure