Towards Flexible Interactive Reflection Removal with Human Guidance

Xiao Chen, Xudong Jiang, Yunkang Tao, Zhen Lei, Qing Li, Chenyang Lei, Zhaoxiang ZhangarXiv 2024 · arXiv · OpenReview

Abstract

Single image reflection removal is inherently ambiguous, as both the reflection and transmission components requiring separation may follow natural image statistics. Existing methods attempt to address the issue by using various types of low-level and physics-based cues as sources of reflection signals. However, these cues are not universally applicable, since they are only observable in specific capture scenarios. This leads to a significant performance drop when test images do not align with their assumptions. In this paper, we aim to explore a novel flexible interactive reflection removal approach that leverages various forms of sparse human guidance, such as points and bounding boxes, as auxiliary high-level prior to achieve robust reflection removal. However, incorporating the raw user guidance naively into the existing reflection removal network does not result in performance gains. To this end, we innovatively transform raw user input into a unified form -- reflection masks using an Interactive Segmentation Foundation Model. Such a design absorbs the quintessence of the foundational segmentation model and flexible human guidance, thereby mitigating the challenges of reflection separations. Furthermore, to fully utilize user guidance and reduce user annotation costs, we design a mask-guided reflection removal network, comprising our proposed self-adaptive prompt block. This block adaptively incorporates user guidance as anchors and refines transmission features via cross-attention mechanisms. Extensive results on real-world images validate that our method demonstrates state-of-the-art performance on various datasets with the help of flexible and sparse user guidance.

Introduction

Removing reflections from a single image is a classic ill-posed problem in computer vision. Because a single pixel represents a mixture of background transmission and foreground reflection, separating them without additional cues is mathematically ambiguous. Existing automatic methods often rely on physical priors, such as the assumption that reflections are blurrier than the background. However, these priors fail in real-world scenarios where reflections can be perfectly sharp.

Traditional interactive methods attempt to solve this by incorporating human guidance, but they are often rigid—supporting only specific input types like dense strokes—and time-consuming. We introduce FIRM (Flexible Interactive Reflection ReMoval), a framework that accepts diverse, sparse inputs such as single points, bounding boxes, or short text descriptions. As shown in Figure 1, FIRM unifies these inputs into a "contrastive mask," achieving state-of-the-art results while reducing human annotation time by over 90%.

Figure 1: Comparison of (a) traditional guidance-specific methods vs. (b) our flexible approach, and (c) the resulting 10x reduction in annotation time.

Methodology

The FIRM pipeline consists of two primary stages: User Guidance Conversion and Mask-Guided Restoration.

1. User Guidance Conversion (UGC)

To handle diverse inputs, we convert all user guidance into a unified contrastive mask. For text prompts, we utilize a language-based segmentation model. For visual cues (points, boxes, strokes), we developed the Segment Any Reflection Model (SARM). SARM is an adaptation of Meta’s Segment Anything Model (SAM), enhanced with a learnable degradation-invariant token and a feature selection block. This allows the model to accurately delineate reflection boundaries even when they overlap with background objects—a task where the original SAM often struggles.

2. Contrastive Mask-Guided Reflection Removal

The restoration network is built upon the NAFNet architecture, integrated with our proposed Contrastive Guidance Interaction Block (CGIB). The CGIB uses a cross-attention mechanism to adaptively fuse the generated contrastive mask with latent image features. This mechanism treats user guidance as "anchors," directing the network to focus on specific regions for layer separation.

Figure 6: The FIRM architecture, showing the conversion of visual/text guidance into a contrastive mask and its integration into the restoration network via CGIB.

Experimental Results

We evaluated FIRM on standard benchmarks (Real20, SIR2) and a newly contributed FIRM Dataset, which includes four types of raw user guidance and corresponding ground-truth masks.

  • Efficiency: FIRM drastically reduces the average interaction time to just 15 seconds per image, compared to 150–234 seconds for previous stroke-based or point-based interactive methods.
  • Restoration Quality: Our method achieves superior performance in terms of PSNR and SSIM. On the challenging "Wild" subset of the SIR2 dataset, FIRM reached a PSNR of 29.71, a significant improvement over the previous state-of-the-art interactive method (26.20).
  • Qualitative Performance: Visual results show that FIRM is much better at removing "sharp" reflections that typically trick automatic single-image methods into thinking they are part of the background.

Conclusion

FIRM demonstrates that unifying diverse human interactions into a single mask format effectively decouples interaction from restoration. By leveraging a foundation model (SAM) and tailoring it for reflection geometry (SARM), we enable high-quality restoration with minimal user effort. While text-based guidance can occasionally falter with vague descriptions, the overall framework provides a flexible and efficient path forward for real-world photo editing.

Our code and dataset are available at: https://github.com/ShawnChenn/FlexibleReflectionRemoval.

Summary updated

Abstract

Single image reflection removal is inherently ambiguous, as both the reflection and transmission components requiring separation may follow natural image statistics. Existing methods attempt to address the issue by using various types of low-level and physics-based cues as sources of reflection signals. However, these cues are not universally applicable, since they are only observable in specific capture scenarios. This leads to a significant performance drop when test images do not align with their assumptions. In this paper, we aim to explore a novel flexible interactive reflection removal approach that leverages various forms of sparse human guidance, such as points and bounding boxes, as auxiliary high-level prior to achieve robust reflection removal. However, incorporating the raw user guidance naively into the existing reflection removal network does not result in performance gains. To this end, we innovatively transform raw user input into a unified form -- reflection masks using an Interactive Segmentation Foundation Model. Such a design absorbs the quintessence of the foundational segmentation model and flexible human guidance, thereby mitigating the challenges of reflection separations. Furthermore, to fully utilize user guidance and reduce user annotation costs, we design a mask-guided reflection removal network, comprising our proposed self-adaptive prompt block. This block adaptively incorporates user guidance as anchors and refines transmission features via cross-attention mechanisms. Extensive results on real-world images validate that our method demonstrates state-of-the-art performance on various datasets with the help of flexible and sparse user guidance. Our code and dataset will be publicly available here https://github.com/ShawnChenn/FlexibleReflectionRemoval.
BibTeX
@misc{chen2024towards,
  author = {Xiao Chen and Xudong Jiang and Yunkang Tao and Zhen Lei and Qing Li and Chenyang Lei and Zhaoxiang Zhang},
  title = {{Towards Flexible Interactive Reflection Removal with Human Guidance}},
  year = {2024},
  eprint = {2406.01555},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.01555},
}
Figure