TL;DR
The authors introduce LoLI-Street, a massive new benchmark dataset containing 34,000 street-scene images designed to advance low-light image enhancement (LLIE) specifically for autonomous driving and surveillance. Alongside this dataset, they propose TriFuse, a novel framework that fuses vision transformers with diffusion models to deliver highly realistic image enhancements in just a fraction of the sampling steps required by traditional diffusion methods.
What Problem Does This Paper Address?
Low-light image enhancement (LLIE) is a critical pre-processing step for downstream computer vision tasks like object detection, tracking, and semantic segmentation. While autonomous vehicles rely heavily on clear vision, they frequently struggle in dark or underexposed real-world environments.
Existing LLIE datasets are limited; they are either small, synthesized, or focused primarily on indoor or generic outdoor scenes. They lack the specific complexities of urban street scenes—such as glare from headlights, deep shadows, and specific object classes like traffic lights and pedestrians. Furthermore, current state-of-the-art (SOTA) models based on diffusion processes or transformers often suffer from a trade-off: they either produce overly bright, unrealistic images or require massive computational overhead (hundreds of sequential denoising steps), making them impractical for real-time deployment.
Methods
The authors propose TriFuse, a framework that elegantly combines the generative power of diffusion models with the long-range dependency modeling of vision transformers.
Low-Light Image ➔ 2D-DWT ➔ [Low-Freq] ➔ Conditional Noise Module (Transformer) ➔ Diffusion Denoising
➔ [High-Freq] ➔ Edge Sharpening Module (CNN + Attention) ➔ Enhanced Image
The architecture consists of three core pillars:
- Discrete Wavelet Transformation (DWT): Instead of processing the raw RGB image directly, TriFuse uses a 2D-DWT (with Haar wavelets) to decompose the low-light image into low-frequency (global structure) and high-frequency (edges and textures) sub-bands.
- Conditional Noise Module (CNM): Traditional diffusion models rely on standard U-Nets to predict noise at each timestep, requiring many iterations. TriFuse replaces this with a custom vision transformer architecture that acts as a highly accurate noise predictor. By conditioning the noise prediction on both the input image and the current timestep, it drastically accelerates the reverse diffusion process.
- Edge Sharpening Module (ESM): To prevent the blurry artifacts common in generative models, the ESM processes the high-frequency sub-bands using depthwise convolutions, Dilated Residual Blocks, and cross-attention mechanisms to align and sharpen structural details.
Datasets
The primary contribution of this work is the LoLI-Street dataset, which consists of 34,000 images partitioned into:
- 30,000 training pairs (synthetic low-light and well-exposed street scenes).
- 3,000 validation pairs.
- 1,000 Real Low-Light Test (RLLT) images captured from high-resolution (4K/8K) videos across various developed cities, manually curated to exclude motion blur.
LoLI-Street categorizes images into three distinct low-light degradation intensities: light, moderate, and dense. It covers 19,000 annotated object classes specifically relevant to autonomous driving.
To evaluate generalization, the authors also test their model on several mainstream datasets, including LOLv1, LOLv2, LSRW, SICE, ExDark, and LLVIP.
Key Insights and Ideas
- Transformers as Diffusion Accelerators: The most compelling insight is that a transformer-based noise predictor (CNM) can capture complex noise patterns far better than standard CNNs. This allows TriFuse to reduce the diffusion process to just 5 sampling steps during inference without sacrificing generation quality.
- Frequency-Domain Specialization: By separating global illumination adjustments (handled via diffusion on low-frequency wavelets) from structural sharpening (handled via the ESM on high-frequency wavelets), the model avoids the common pitfall of artifact generation and color distortion.
- Cross-Dataset Generalization: Models trained on LoLI-Street demonstrate remarkable zero-shot generalization on completely different datasets (like indoor scenes in LOLv1), indicating that street-scene data provides a highly robust and diverse feature space for training.
Key Results
TriFuse was benchmarked against several recent SOTA models, including Retinexformer, DiffLL, and LLFormer.
- Visual Quality: On the Real Low-Light Test (RLLT) set, TriFuse achieved the best no-reference image quality scores, yielding a BRISQUE score of 10.32 and a NIQE score of 10.61 (where lower is better), significantly outperforming competitors which averaged BRISQUE scores between 14 and 41.
- Downstream Object Detection: When evaluating enhanced images using YOLOv10, TriFuse achieved a mAP(0.5) of 0.753 and a mAP(0.5-0.9) of 0.692, representing a massive leap forward for autonomous perception accuracy in the dark.
- Computational Efficiency: Thanks to the 5-step sampling strategy, TriFuse achieves an average inference time of 0.173 seconds, making it orders of magnitude faster than standard diffusion counterparts like DiffLL (1.850s).
Limitations and Future Work
While TriFuse represents a massive speedup for diffusion-based enhancement, an inference time of 0.173 seconds translates to roughly 6 frames per second (FPS). This is still a bottleneck for real-time autonomous vehicle control loops, which typically require at least 30 FPS.
The authors note that future work will focus on further optimizing the network architecture for real-time deployment, adapting the framework to handle multi-adverse weather conditions (such as nighttime fog or rain), and exploring fully unsupervised training paradigms to eliminate the need for paired synthetic data.
Summary updated