Disentangle Nighttime Lens Flares: Self-supervised Generation-based Lens Flare Removal

Yuwen He, Wei Wang, Wanyu Wang, Kui Jiang

Nighttime photography is frequently compromised by lens flares—optical artifacts resulting from light reflection and refraction within camera sensor arrays. These artifacts typically manifest as "glow" (a hazy aura surrounding bright light sources) and "reflective flares" or "ghosting" (distinct geometric patterns elsewhere in the frame). While often treated as separate issues, these flares almost always co-occur and are physically interdependent, originating from the same light source and passing through the same optical system.

Traditional flare removal methods often specialize in a single flare type, failing to address the complex, non-additive relationship between co-existing artifacts. A new approach presented at AAAI 2025, SGLFR-Net, introduces a self-supervised framework designed to disentangle and remove these joint flares by modeling their intrinsic physical relationship.

The Challenge of Joint Flare Removal

Most existing solutions are "black-box" models trained on synthetic data. However, capturing real-world ground-truth pairs (the same scene with and without flares) is nearly impossible. Furthermore, simply cascading individual de-glow and de-ghosting algorithms often results in residual artifacts because it ignores the shared physical origin of the flares.

Methodology: SGLFR-Net

The proposed Self-supervised Generation-based Lens Flare Removal Network (SGLFR-Net) moves away from standard supervised learning. Instead, it leverages a physics-informed model of nighttime lens flare formation. The system consists of two primary components that work in parallel to "re-generate" and then subtract the flare components from the input image.

The SGLFR-Net architecture, featuring PSFR-Net for glow removal and TPRR-Net with OS-TPM for reflective flare removal.

PSFR-Net: Modeling the Glow

The PSF Rendering Network (PSFR-Net) focuses on suppressing glow. It utilizes a Point Spread Function (PSF) rendering prior to model light scattering. By learning to generate a fuzzy kernel that matches the specific glow in the input, it can accurately estimate and remove the hazy aura around light sources.

TPRR-Net and Optical Symmetry

The Texture Prior based Reflection Removal Network (TPRR-Net) handles reflective flares. It employs a novel Optical Symmetry based Texture Prior (OS-TPM). This module exploits the physical property that reflective flares often appear symmetrically across the optical center from their source. By identifying these symmetric regions, the network can effectively mask and "fill in" ghosting artifacts using texture priors.

Because SGLFR-Net is self-supervised, it optimizes its weights for each specific image during inference, requiring no pre-training on large datasets.

Results and Downstream Impact

Evaluations on synthetic benchmarks (OurSynDatasets, Flare7K) and real-world images demonstrate that SGLFR-Net outperforms state-of-the-art (SOTA) cascaded methods. Quantitatively, it achieved a PSNR of 29.65 and an SSIM of 0.974 on synthetic joint-flare datasets.

Qualitative comparison of flare removal methods and their impact on downstream semantic segmentation.

Beyond aesthetic improvements, the paper highlights the benefits of flare removal for computer vision tasks. As shown in the segmentation results, removing flares significantly improves the accuracy of semantic segmentation models, allowing them to correctly identify objects like trees and buildings that were previously obscured by light streaks.

Conclusion and Future Directions

SGLFR-Net represents a significant step toward robust nighttime image restoration. By grounding the architecture in optical physics and leveraging self-supervision, the researchers have created a tool that works "out of the box" for various camera sensors. Future work may focus on reducing the computational overhead of per-image optimization to enable real-time applications and extending the model to handle the more complex lighting conditions found in daytime photography.

This summary is based on the paper "Disentangle Nighttime Lens Flares: Self-supervised Generation-based Lens Flare Removal," presented at AAAI 2025.

Summary updated

Abstract

Lens flares arise from light reflection and refraction within sensor arrays, whose diverse types include glow, veiling glare, reflective flare and so on. Existing methods are specialized for one specific type only, and overlook the simultaneous occurrence of multiple typed lens flares, which is common in the real-world, e.g. coexistence of glow and displacement reflections from the same light source. These co-occurring lens flares cannot be effectively resolved by the simple combination of individual flare removal methods, since these coexisting flares originates from the same light source and are generated simultaneously within the same sensor array, exhibit a complex interdependence rather than simple additive relation. To model this interdependent flare relationship, our Nighttime Lens Flare Formation model is the first attempt to learn the intrinsic physical relationship between flares on the imaging plane. Building on this physical model, we introduce a solution to this joint flare removal task named Self-supervised Generation-based Lens Flare Removal Network (SGLFR-Net), which is self-supervised without pre-training. Specifically, the nighttime glow is detangled in PSF Rendering Network(PSFR-Net) based on PSF Rendering Prior, while the reflective flare is modelled in Texture Prior Based Reflection Flare Removal Network (TPRR-Net). Empirical evaluations demonstrate the effectiveness of the proposed method in both joint and individual glare removal tasks.
BibTeX
@inproceedings{he2025disentangle,
  author = {Yuwen He and Wei Wang and Wanyu Wang and Kui Jiang},
  title = {{Disentangle Nighttime Lens Flares: Self-supervised Generation-based Lens Flare Removal}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i3.32359},
}
Figure