TL;DR
This paper introduces a more realistic lens flare removal framework by rethinking how synthetic training data is generated. By accounting for the non-linear effects of Image Signal Processing (ISP) pipelines—specifically auto-exposure and tone mapping—the authors create a synthesis pipeline that better mimics real-world photography, alongside a novel "soft" recovery strategy to preserve multiple light sources without using arbitrary thresholds.
What Problem Does This Paper Address?
Lens flare is a common optical artifact caused by light scattering or reflecting within a lens system. While annoying for photographers, it’s a major hurdle for computer vision tasks like object detection in autonomous driving.
The core challenge in flare removal is the lack of "paired" data (the same scene with and without flare). To solve this, researchers usually synthesize data by taking a clean image and adding a flare layer. However, most existing methods use a "direct-add" approach in RAW or gamma-inversed space. The authors argue this is fundamentally flawed because it ignores the Image Signal Processing (ISP) pipeline. In reality, a camera's Auto-Exposure (AE) will darken a scene when a bright light source is present, and Tone Mapping (TMO) will non-linearly compress the highlights. Ignoring these leads to a "distribution shift," where models trained on synthetic data fail to generalize to real-world photos captured by consumer electronics.
Furthermore, existing methods often accidentally remove the light source itself or struggle to handle multiple light sources (like streetlights at night) because they rely on hard intensity thresholds to identify what to keep.
Methods
The authors propose two major innovations:
-
ISP-Aware Synthesis Pipeline: Instead of simply adding flare to a scene ($I = S + F$), they use a pixel-wise convex combination. They calculate an illuminance matrix for the flare and use a sigmoid-based weight function to blend the scene and flare. This approach naturally simulates how a bright flare "occludes" background details and how the overall intensity distribution shifts, avoiding the over-saturation and contrast loss typical of the direct-add method.
Figure 2. Comparison of the flare-corrupted image and light source recovery in previous works and our method. Our method can synthesize a more realistic flare-corrupted image and preserve more natural light sources. -
Multiple Light Source Recovery: To ensure light sources aren't erased, the authors implement a recovery strategy that convexly averages the neural network's input and output. They use a strong convex weight function ($x^\alpha$) that assigns higher weights to pixels in the "Shoulder section" of the tone mapping curve (the brightest parts). This "soft" approach allows the model to preserve multiple light sources of varying sizes and shapes without needing a hard-coded brightness threshold.
Datasets
The study utilizes several datasets for training and benchmarking:
- Flare7K & Wu et al.: Existing datasets of flare-only images used for synthesis.
- Consumer Electronics Test Dataset (New): The authors contributed a new benchmark consisting of 100 real-world flare-corrupted images captured by ten different devices, including the iPhone 13 Pro, Huawei Mate 20, and iPad Air 4. This dataset is designed specifically to test the generalization capability of flare removal models across diverse hardware.
Key Insights and Ideas
- The "Direct-Add" Fallacy: Adding flare to an image in digital space isn't how physics works in a camera. Because of Auto-Exposure, the presence of a bright light source actually changes how the rest of the scene is captured.
- Convex Combination is Key: By treating the blending as a convex combination ($I = (1-W)S + WF$), the model learns that bright flare regions should dominate the background, which aligns with real-world imaging physics.
- Avoiding Hard Thresholds: Light source recovery shouldn't be a binary "keep or delete" decision based on a 0.99 brightness cutoff. Using a continuous power function ($\alpha=15$) allows for a natural transition that preserves the glow and shape of light sources like the moon or distant lamps.
Key Results
The proposed pipeline significantly outperforms previous state-of-the-art methods both quantitatively and qualitatively:
- Quantitative: On the Flare7K test set, their method combined with a Uformer backbone achieved a PSNR of 26.3, compared to 23.7 for the previous leading method (Wu et al.).
- Qualitative: As shown in Figure 1, the model effectively removes complex streaks and halos while keeping the primary light source intact across various mobile devices.
- Downstream Tasks: The authors demonstrated that their flare removal significantly improves object detection. In flare-corrupted images, detectors often mistake flares for cars or fail to see objects shaded by streaks; removing the flare with this method restored the detector's accuracy.
Limitations and Future Work
While the paper demonstrates strong generalization, it primarily focuses on the synthesis and recovery logic rather than new network architectures (using existing backbones like U-Net and Uformer). The authors note that while their method handles multiple light sources well, extremely complex nighttime scenes with dozens of small, overlapping light sources remain a challenge. Future work could involve more sophisticated modeling of specific lens coatings or integrating the flare removal directly into the camera's ISP hardware.
Summary updated