TL;DR
This paper introduces a novel multi-stage deep learning approach for removing reflections from images captured in a burst sequence. By leveraging subtle motion differences between the reflection and transmission layers caused by camera shake, their method, called Reflection Motion Aggregation (RMA), effectively separates and removes reflections, achieving state-of-the-art results with significantly improved efficiency.
What Problem Does This Paper Address?
Reflections in images, particularly from glass surfaces, are a common and persistent problem that degrades image quality. While single-image reflection removal methods have seen progress, they often struggle with complex reflections. Multi-image approaches offer better performance by exploiting differences across multiple views, but typically require users to capture images with significant camera movement and precise alignment, which is inconvenient. This paper addresses the need for a reflection removal method that combines the convenience of single-image capture (a quick burst of photos) with the superior performance of multi-image techniques, specifically by utilizing the natural handshakes that occur during burst photography.
Methods
The proposed method is a three-stage deep learning pipeline:
- Single Image Reflection Suppression (SIRS): An initial stage that suppresses reflections in each individual image of the burst. This uses a modified lightweight network architecture and helps in subsequent alignment steps.
- Reflection Motion Aggregation (RMA): This is the core novel contribution. After aligning the burst images (using homography estimation aided by the SIRS stage), this stage extracts a "Reflection Motion Aggregation" cue. The idea is that reflections and the underlying transmission layer move differently due to camera shake. The RMA cue aggregates the motion of the reflection layer, de-emphasizing it relative to the transmission layer, thereby creating a clearer distinction between the two.
- Guided Reflection Removal: The final stage uses the generated RMA cue to guide a deep network in removing the remaining reflections from a reference image (typically the middle frame of the burst).
The training process involves a combination of contextual and pixel losses, with specific strategies to handle potential misalignments in the burst sequences.
Datasets
The authors introduce a new dataset and utilize existing ones:
- Synthetic Burst Reflection (SBR) dataset: A synthetic dataset generated by simulating burst captures with reflections, based on planar scene assumptions and motion models derived from handshake patterns.
- Real World Burst Image Reflection Removal (BIRR) dataset (Ours): A newly captured dataset comprising over 1000 sets of burst images. It's split into:
- Aligned Dataset: Captured using a tripod for perfect alignment between input bursts and ground truth transmission. Primarily used for training.
- Hand-Held Dataset: Captured using handheld smartphones, where input bursts are not perfectly aligned with the ground truth. Used for testing.
- Large Baseline Dataset: A publicly available dataset [32] with videos captured by moving a camera to create large viewpoint differences, used to evaluate the method's performance on non-burst multi-image scenarios.
Key Insights and Ideas
- Leveraging Camera Shake: The most significant insight is the realization that the subtle, uncontrolled motion from camera shake during a burst capture provides valuable cues for separating reflection and transmission layers. This turns a potential problem into a feature.
- Reflection Motion Aggregation (RMA) Cue: The novel RMA cue effectively highlights the differences in motion between reflection and transmission layers, creating a powerful guide for the final reflection removal stage.
- Convenience Meets Performance: The paper successfully bridges the gap between user-friendly single-image capture and high-performance multi-image reflection removal.
Key Results
- Quantitative Improvements: The proposed method achieves significant improvements over state-of-the-art methods:
- ~2dB improvement in PSNR over single-image based methods.
- ~1dB improvement over multi-image based methods on the SBR and BIRR datasets.
- ~2dB improvement on the Large Baseline dataset.
- Qualitative Results: The method produces aesthetically pleasing results, effectively suppressing strong reflections and yielding cleaner outputs compared to existing methods.
- Efficiency: The method is significantly faster than complex online optimization-based multi-image methods, making it more suitable for real-time applications.
Limitations and Future Work
- Assumptions: The synthetic dataset generation relies on assumptions of planar scenes and specific motion models. While the real-world dataset mitigates this, the synthetic training might not perfectly capture all real-world complexities.
- Alignment Errors: While the method is designed to handle some misalignments, severe misalignments could still pose challenges.
- Future Work: The paper does not explicitly detail future work, but the success of the RMA cue suggests potential applications in other layer separation tasks where subtle motion differences exist. The release of the BIRR dataset will also enable further research.
Summary updated