TL;DR
This paper introduces a novel self-supervised method, SGLFR-Net, to tackle the challenging problem of simultaneously removing both glow and reflective lens flares in nighttime images. By proposing a physical model for joint flare formation and leveraging two distinct priors (PSF rendering for glow and optical symmetry for reflection), their approach achieves state-of-the-art results without requiring any pre-training data.
What Problem Does This Paper Address?
Nighttime photography often suffers from lens flares, which are optical artifacts caused by light scattering and reflection within camera lenses. These flares can obscure important details and hinder downstream applications. Existing methods often focus on specific types of flares (e.g., glow or reflective flares) and struggle to handle their simultaneous occurrence, which is common in real-world scenarios. Furthermore, these methods often rely on large, paired datasets for supervised training, which are difficult to obtain for nighttime lens flares due to the intricate nature of their formation. This paper addresses the challenge of jointly removing both glow and reflective lens flares in a self-supervised manner, without the need for pre-training.
Methods
The core of the proposed solution is the Self-supervised Generation-based Lens Flare Removal Network (SGLFR-Net). This network is built upon a novel Nighttime Lens Flare Formation model that physically describes how glow and reflective flares are jointly generated. This model leads to two key priors:
- PSF Rendering Prior: This prior is used to model the glow flare, which arises from light scattering. The PSF Rendering Network (PSFR-Net) stream within SGLFR-Net simulates this glow formation using a PSF rendering approach.
- Optical Symmetry Prior: This prior leverages the observation that nighttime reflective flares (ghosts) exhibit optical symmetry when related to the light source. The Texture Prior Based Reflection Removal Network (TPRR-Net) stream, guided by the Optical Symmetry Based Texture Prior Module (OS-TPM), uses this prior to model and remove reflective flares.
The SGLFR-Net is designed as a two-stream network. PSFR-Net generates a glow-rendered output, while TPRR-Net produces an intermediate result that is then refined using the OS-TPM. The entire process is self-supervised, meaning it learns without explicit ground truth flare masks during training.
Datasets
The paper utilizes the following datasets:
- OurSynDatasets: A self-synthesized dataset for joint glow and ghost flare removal, created from existing datasets and processed to simulate nighttime flares.
- BracketFlare Dataset: Used for evaluating reflective flare removal.
- ECCV2022 Dataset: Used for evaluating glow suppression tasks.
- Real-world captured datasets: Used for evaluating all tasks.
The authors note that the OurSynDatasets were stimulated from images in a training dataset from BracketFlare (Dai, Luo et al. 2023) and processed with a function y~U(1.4,1.8), as there were no readily available joint flare datasets.
Key Insights and Ideas
- Unified Flare Formation Model: The paper's most significant contribution is the proposal of a physical model that unifies the formation of both glow and reflective lens flares. This is crucial because these flares are not independent but rather arise from the same light source and imaging process.
- Self-Supervised Learning for Joint Flare Removal: The development of a self-supervised framework for joint flare removal is a major step forward, as it bypasses the need for expensive and difficult-to-obtain paired training data.
- Leveraging Distinct Priors: The successful integration of two distinct priors – PSF rendering for glow and optical symmetry for reflection – into a single network highlights the power of combining different physical intuitions for complex image restoration tasks.
- Improved Semantic Segmentation: The paper demonstrates that by effectively removing lens flares, their method also improves the results of semantic segmentation tasks on the same images, indicating a more accurate recovery of the underlying scene.
Key Results
- State-of-the-Art Performance: The proposed SGLFR-Net achieves state-of-the-art results on the joint glow and reflective/ghost flare removal task, outperforming existing methods by significant margins in PSNR and SSIM (e.g., 1.4% and 0.4% advantages respectively on their SynDatasets).
- Effective Individual Flare Removal: The method also performs well on individual glow suppression and reflective flare removal tasks, often surpassing specialized methods.
- No Pre-training Required: A key advantage is that the method is self-supervised and requires no pre-training, making it more adaptable and easier to deploy.
- Qualitative Improvements: Visual comparisons show that SGLFR-Net effectively removes both types of flares, recovering details around light sources and producing cleaner images compared to existing methods, especially in challenging scenarios with co-occurring flares.
Limitations and Future Work
- Limitations: The paper acknowledges that while their method is self-supervised, it relies on the accuracy of the proposed physical model. The effectiveness might be limited in scenarios where the flare formation deviates significantly from this model. Additionally, the paper mentions that more joint task results with unreported combinations can be found in their supplementary material, suggesting that a comprehensive comparison across all possible combinations might be extensive.
- Future Work: The authors suggest that their work contributes to a better understanding of lens flare formation and removal. Future research could explore more complex flare types or investigate real-time applications of their method. The paper also implicitly suggests that further refinement of the physical models and priors could lead to even more robust solutions.
Summary updated