Learning to Jointly Generate and Separate Reflections

Daiqian Ma, Renjie Wan, Boxin Shi, Alex C. Kot, Lingyu Duan

Abstract

Single image reflection removal (SIRR) is a challenging ill-posed problem in computer vision. Traditional deep learning approaches rely on synthetic data or perfectly aligned real-world triplets, both of which have significant drawbacks in capturing real-world complexity or scaling. This paper proposes a weakly-supervised framework that jointly learns to generate and separate reflections using a large-scale unpaired dataset. By employing an entanglement-disentanglement mechanism and cycle consistency, the model achieves state-of-the-art performance on real-world benchmarks while remaining highly efficient.

Introduction

Removing reflections from a single image is a classic problem where a mixture image $M$ must be decomposed into a background layer $B$ and a reflection layer $R$. Existing methods struggle with two main issues: the scarcity of perfectly registered training triplets and the "linear trap," where synthetic data fails to capture the complex physics of real-world reflections, such as non-uniform lighting and ghosting artifacts.

Overview of synthetic paired data vs. the proposed real-world unpaired data pipeline.

To address these, the authors propose a framework that leverages abundant unpaired data. Instead of relying on simple linear combinations, the model learns the underlying distribution of reflections through a joint generation and separation process, bridging the gap between synthetic training and real-world application.

Methodology

The framework consists of two primary components: a Generator ($G$) and a Separator ($S$), which work together in a cycle-consistent manner.

Joint Generation and Separation

The Generator ($G$) performs "entanglement," learning to fuse a random background and reflection into a realistic mixture. Unlike simple linear addition, $G$ is trained to model the complex physical properties of real reflections. The Separator ($S$) performs "disentanglement," splitting a mixture into its constituent background, reflection, and an auxiliary edge map.

The proposed cycle-consistent framework showing (a) the generation-to-separation cycle and (b) the separation-to-generation cycle.

Cycle Consistency and Multi-Task Learning

As shown in the architecture diagram, the model utilizes two cycles to enforce consistency without requiring paired ground truth:

  1. Generation-Separation Cycle: This cycle starts with separate background ($B$) and reflection ($R$) images. $G$ creates a synthetic mixture $M^*$, which $S$ then attempts to decompose back into $B^*$ and $R^*$. This ensures that the generation process produces mixtures that are actually separable.
  2. Separation-Generation Cycle: This cycle starts with a real-world mixture $M$. $S$ separates it into $B^*$ and $R^*$, which $G$ then uses to reconstruct the original mixture $M^*$. This ensures the separator captures all relevant information from the real input.

To improve detail preservation, the separator also predicts an edge map $E$. This multi-task approach helps the model distinguish between sharp background features and typically blurrier reflection artifacts. The training is guided by a combination of Adversarial loss (to ensure visual realism), Pixel reconstruction loss ($L_1$), and Content reconstruction loss using VGG feature maps.

Datasets and Experiments

The authors introduced a new Unpaired Reflection Dataset containing 4,027 images of mixtures, backgrounds, and reflections collected across various scenes and lighting conditions. This dataset provides the variety needed for the model to generalize to diverse real-world scenarios.

Evaluation was performed on the standard SIR$^2$ Benchmark. The proposed method achieved a PSNR of 24.48 and an SSIM of 0.903, surpassing previous state-of-the-art models. Furthermore, the model is highly efficient, processing a $224 \times 288$ image in just 0.0214 seconds, making it roughly 15x faster than some competing deep learning approaches.

Conclusion

By treating reflection generation and separation as dual tasks, this framework overcomes the limitations of synthetic and paired data. The use of weak supervision and edge-aware multi-task learning allows for robust, high-quality reflection removal in real-world images. While extreme reflections remain a challenge, the authors suggest that this joint learning paradigm could be extended to other restoration tasks like deraining or dehazing.

Summary updated

Abstract

Existing learning-based single image reflection removal methods using paired training data have fundamental limitations about the generalization capability on real-world reflections due to the limited variations in training pairs. In this work, we propose to jointly generate and separate reflections within a weakly-supervised learning framework, aiming to model the reflection image formation more comprehensively with abundant unpaired supervision. By imposing the adversarial losses and combinable mapping mechanism in a multi-task structure, the proposed framework elegantly integrates the two separate stages of reflection generation and separation into a unified model. The gradient constraint is incorporated into the concurrent training process of the multi-task learning as well. In particular, we built up an unpaired reflection dataset with 4,027 images, which is useful for facilitating the weakly-supervised learning of reflection removal model. Extensive experiments on a public benchmark dataset show that our framework performs favorably against state-of-the-art methods and consistently produces visually appealing results.
BibTeX
@inproceedings{ma2019learning,
  author = {Daiqian Ma and Renjie Wan and Boxin Shi and Alex C. Kot and Lingyu Duan},
  title = {{Learning to Jointly Generate and Separate Reflections}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00253},
}
Figure