Disentangle Nighttime Lens Flares: Self-supervised Generation-based Lens Flare Removal

Yuwen He, Wei Wang, Wanyu Wu, Kui Jiang

TL;DR

This paper introduces a novel self-supervised method, SGLFR-Net, to tackle the challenging problem of simultaneously removing both glow and reflective lens flares in nighttime images. By proposing a physical model for joint flare formation and leveraging two distinct priors (PSF rendering for glow and optical symmetry for reflection), their approach achieves state-of-the-art results without requiring any pre-training data.

What Problem Does This Paper Address?

Nighttime photography often suffers from lens flares, which are optical artifacts caused by light scattering and reflection within camera lenses. These flares can obscure important details and hinder downstream applications. Existing methods often focus on specific types of flares (e.g., glow or reflective flares) and struggle to handle their simultaneous occurrence, which is common in real-world scenarios. Furthermore, these methods often rely on large, paired datasets for supervised training, which are difficult to obtain for nighttime lens flares due to the intricate nature of their formation. This paper addresses the challenge of jointly removing both glow and reflective lens flares in a self-supervised manner, without the need for pre-training.

Methods

The core of the proposed solution is the Self-supervised Generation-based Lens Flare Removal Network (SGLFR-Net). This network is built upon a novel Nighttime Lens Flare Formation model that physically describes how glow and reflective flares are jointly generated. This model leads to two key priors:

  • PSF Rendering Prior: This prior is used to model the glow flare, which arises from light scattering. The PSF Rendering Network (PSFR-Net) stream within SGLFR-Net simulates this glow formation using a PSF rendering approach.
  • Optical Symmetry Prior: This prior leverages the observation that nighttime reflective flares (ghosts) exhibit optical symmetry when related to the light source. The Texture Prior Based Reflection Removal Network (TPRR-Net) stream, guided by the Optical Symmetry Based Texture Prior Module (OS-TPM), uses this prior to model and remove reflective flares.

The SGLFR-Net is designed as a two-stream network. PSFR-Net generates a glow-rendered output, while TPRR-Net produces an intermediate result that is then refined using the OS-TPM. The entire process is self-supervised, meaning it learns without explicit ground truth flare masks during training.

Datasets

The paper utilizes the following datasets:

  • OurSynDatasets: A self-synthesized dataset for joint glow and ghost flare removal, created from existing datasets and processed to simulate nighttime flares.
  • BracketFlare Dataset: Used for evaluating reflective flare removal.
  • ECCV2022 Dataset: Used for evaluating glow suppression tasks.
  • Real-world captured datasets: Used for evaluating all tasks.

The authors note that the OurSynDatasets were stimulated from images in a training dataset from BracketFlare (Dai, Luo et al. 2023) and processed with a function y~U(1.4,1.8), as there were no readily available joint flare datasets.

Key Insights and Ideas

  • Unified Flare Formation Model: The paper's most significant contribution is the proposal of a physical model that unifies the formation of both glow and reflective lens flares. This is crucial because these flares are not independent but rather arise from the same light source and imaging process.
  • Self-Supervised Learning for Joint Flare Removal: The development of a self-supervised framework for joint flare removal is a major step forward, as it bypasses the need for expensive and difficult-to-obtain paired training data.
  • Leveraging Distinct Priors: The successful integration of two distinct priors – PSF rendering for glow and optical symmetry for reflection – into a single network highlights the power of combining different physical intuitions for complex image restoration tasks.
  • Improved Semantic Segmentation: The paper demonstrates that by effectively removing lens flares, their method also improves the results of semantic segmentation tasks on the same images, indicating a more accurate recovery of the underlying scene.

Key Results

  • State-of-the-Art Performance: The proposed SGLFR-Net achieves state-of-the-art results on the joint glow and reflective/ghost flare removal task, outperforming existing methods by significant margins in PSNR and SSIM (e.g., 1.4% and 0.4% advantages respectively on their SynDatasets).
  • Effective Individual Flare Removal: The method also performs well on individual glow suppression and reflective flare removal tasks, often surpassing specialized methods.
  • No Pre-training Required: A key advantage is that the method is self-supervised and requires no pre-training, making it more adaptable and easier to deploy.
  • Qualitative Improvements: Visual comparisons show that SGLFR-Net effectively removes both types of flares, recovering details around light sources and producing cleaner images compared to existing methods, especially in challenging scenarios with co-occurring flares.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method is self-supervised, it relies on the accuracy of the proposed physical model. The effectiveness might be limited in scenarios where the flare formation deviates significantly from this model. Additionally, the paper mentions that more joint task results with unreported combinations can be found in their supplementary material, suggesting that a comprehensive comparison across all possible combinations might be extensive.
  • Future Work: The authors suggest that their work contributes to a better understanding of lens flare formation and removal. Future research could explore more complex flare types or investigate real-time applications of their method. The paper also implicitly suggests that further refinement of the physical models and priors could lead to even more robust solutions.

Summary updated

Abstract

Lens flares arise from light reflection and refraction within sensor arrays, whose diverse types include glow, veiling glare, reflective flare and so on. Existing methods are specialized for one specific type only, and overlook the simultaneous occurrence of multiple typed lens flares, which is common in the real-world, e.g. coexistence of glow and displacement reflections from the same light source. These co-occurring lens flares cannot be effectively resolved by the simple combination of individual flare removal methods, since these coexisting flares originates from the same light source and are generated simultaneously within the same sensor array, exhibit a complex interdependence rather than simple additive relation. To model this interdependent flare relationship, our Nighttime Lens Flare Formation model is the first attempt to learn the intrinsic physical relationship between flares on the imaging plane. Building on this physical model, we introduce a solution to this joint flare removal task named Self-supervised Generation-based Lens Flare Removal Network (SGLFR-Net), which is self-supervised without pre-training. Specifically, the nighttime glow is detangled in PSF Rendering Network(PSFR-Net) based on PSF Rendering Prior, while the reflective flare is modelled in Texture Prior Based Reflection Flare Removal Network (TPRR-Net). Empirical evaluations demonstrate the effectiveness of the proposed method in both joint and individual glare removal tasks.
BibTeX
@inproceedings{he2025disentangle,
  author = {Yuwen He and Wei Wang and Wanyu Wu and Kui Jiang},
  title = {{Disentangle Nighttime Lens Flares: Self-supervised Generation-based Lens Flare Removal}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i3.32359},
}
Figure