LoLI-Street: Benchmarking Low-Light Image Enhancement and Beyond

Md Tanvir Islam, Inzamamul Alam, Simon S. Woo, Saeed Anwar, Ik Hyun Lee, Khan Muhammad

TL;DR

The authors introduce LoLI-Street, a massive new benchmark dataset containing 34,000 street-scene images designed to advance low-light image enhancement (LLIE) specifically for autonomous driving and surveillance. Alongside this dataset, they propose TriFuse, a novel framework that fuses vision transformers with diffusion models to deliver highly realistic image enhancements in just a fraction of the sampling steps required by traditional diffusion methods.

What Problem Does This Paper Address?

Low-light image enhancement (LLIE) is a critical pre-processing step for downstream computer vision tasks like object detection, tracking, and semantic segmentation. While autonomous vehicles rely heavily on clear vision, they frequently struggle in dark or underexposed real-world environments.

Existing LLIE datasets are limited; they are either small, synthesized, or focused primarily on indoor or generic outdoor scenes. They lack the specific complexities of urban street scenes—such as glare from headlights, deep shadows, and specific object classes like traffic lights and pedestrians. Furthermore, current state-of-the-art (SOTA) models based on diffusion processes or transformers often suffer from a trade-off: they either produce overly bright, unrealistic images or require massive computational overhead (hundreds of sequential denoising steps), making them impractical for real-time deployment.

Methods

The authors propose TriFuse, a framework that elegantly combines the generative power of diffusion models with the long-range dependency modeling of vision transformers.

Low-Light Image ➔ 2D-DWT ➔ [Low-Freq]  ➔ Conditional Noise Module (Transformer) ➔ Diffusion Denoising
                         ➔ [High-Freq] ➔ Edge Sharpening Module (CNN + Attention) ➔ Enhanced Image

The architecture consists of three core pillars:

  1. Discrete Wavelet Transformation (DWT): Instead of processing the raw RGB image directly, TriFuse uses a 2D-DWT (with Haar wavelets) to decompose the low-light image into low-frequency (global structure) and high-frequency (edges and textures) sub-bands.
  2. Conditional Noise Module (CNM): Traditional diffusion models rely on standard U-Nets to predict noise at each timestep, requiring many iterations. TriFuse replaces this with a custom vision transformer architecture that acts as a highly accurate noise predictor. By conditioning the noise prediction on both the input image and the current timestep, it drastically accelerates the reverse diffusion process.
  3. Edge Sharpening Module (ESM): To prevent the blurry artifacts common in generative models, the ESM processes the high-frequency sub-bands using depthwise convolutions, Dilated Residual Blocks, and cross-attention mechanisms to align and sharpen structural details.

Datasets

The primary contribution of this work is the LoLI-Street dataset, which consists of 34,000 images partitioned into:

  • 30,000 training pairs (synthetic low-light and well-exposed street scenes).
  • 3,000 validation pairs.
  • 1,000 Real Low-Light Test (RLLT) images captured from high-resolution (4K/8K) videos across various developed cities, manually curated to exclude motion blur.

LoLI-Street categorizes images into three distinct low-light degradation intensities: light, moderate, and dense. It covers 19,000 annotated object classes specifically relevant to autonomous driving.

To evaluate generalization, the authors also test their model on several mainstream datasets, including LOLv1, LOLv2, LSRW, SICE, ExDark, and LLVIP.

Sample images from the LoLI-Street dataset showing different degradation levels and real-world test cases.

Key Insights and Ideas

  • Transformers as Diffusion Accelerators: The most compelling insight is that a transformer-based noise predictor (CNM) can capture complex noise patterns far better than standard CNNs. This allows TriFuse to reduce the diffusion process to just 5 sampling steps during inference without sacrificing generation quality.
  • Frequency-Domain Specialization: By separating global illumination adjustments (handled via diffusion on low-frequency wavelets) from structural sharpening (handled via the ESM on high-frequency wavelets), the model avoids the common pitfall of artifact generation and color distortion.
  • Cross-Dataset Generalization: Models trained on LoLI-Street demonstrate remarkable zero-shot generalization on completely different datasets (like indoor scenes in LOLv1), indicating that street-scene data provides a highly robust and diverse feature space for training.

Key Results

TriFuse was benchmarked against several recent SOTA models, including Retinexformer, DiffLL, and LLFormer.

  • Visual Quality: On the Real Low-Light Test (RLLT) set, TriFuse achieved the best no-reference image quality scores, yielding a BRISQUE score of 10.32 and a NIQE score of 10.61 (where lower is better), significantly outperforming competitors which averaged BRISQUE scores between 14 and 41.
  • Downstream Object Detection: When evaluating enhanced images using YOLOv10, TriFuse achieved a mAP(0.5) of 0.753 and a mAP(0.5-0.9) of 0.692, representing a massive leap forward for autonomous perception accuracy in the dark.
  • Computational Efficiency: Thanks to the 5-step sampling strategy, TriFuse achieves an average inference time of 0.173 seconds, making it orders of magnitude faster than standard diffusion counterparts like DiffLL (1.850s).
Fig. 1: Comparison between our TriFuse and SOTA models using a sample real low-light test image. (a) Qualitative comparison: Visually, PairLIE and RQ- LLIE produce brighter outputs but lack realism. In contrast, TriFuse ensures high visual quality with realistic enhancements. (b) Quantitative comparison based on the no-reference metric BRISQUE (↓) and inference time (↓).

Limitations and Future Work

While TriFuse represents a massive speedup for diffusion-based enhancement, an inference time of 0.173 seconds translates to roughly 6 frames per second (FPS). This is still a bottleneck for real-time autonomous vehicle control loops, which typically require at least 30 FPS.

The authors note that future work will focus on further optimizing the network architecture for real-time deployment, adapting the framework to handle multi-adverse weather conditions (such as nighttime fog or rain), and exploring fully unsupervised training paradigms to eliminate the need for paired synthetic data.

Summary updated

Abstract

Low-light image enhancement (LLIE) is essential for numerous computer vision tasks, including object detection, tracking, segmentation, and scene understanding. Despite substantial research on improving low-quality images captured in underexposed conditions, clear vision remains critical for autonomous vehicles, which often struggle with low-light scenarios, signifying the need for continuous research. However, paired datasets for LLIE are scarce, particularly for street scenes, limiting the development of robust LLIE methods. Despite using advanced transformers and/or diffusion-based models, current LLIE methods struggle in real-world low-light conditions and lack training on street-scene datasets, limiting their effectiveness for autonomous vehicles. To bridge these gaps, we introduce a new dataset LoLI-Street (Low-Light Images of Streets) with 33k paired low-light and well-exposed images from street scenes in developed cities, covering 19k object classes for object detection. LoLI-Street dataset also features 1,000 real low-light test images for testing LLIE models under real-life conditions. Furthermore, we propose a transformer and diffusion-based LLIE model named "TriFuse". Leveraging the LoLI-Street dataset, we train and evaluate our TriFuse and SOTA models to benchmark on our dataset. Comparing various models, our dataset's generalization feasibility is evident in testing across different mainstream datasets by significantly enhancing images and object detection for practical applications in autonomous driving and surveillance systems. The complete code and dataset is available on https://github.com/tanvirnwu/TriFuse.
BibTeX
@inproceedings{islam2024loli,
  author = {Md Tanvir Islam and Inzamamul Alam and Simon S. Woo and Saeed Anwar and IK Hyun Lee and Khan Muhammad},
  title = {{LoLI-Street: Benchmarking Low-Light Image Enhancement and Beyond}},
  booktitle = {ACCV},
  year = {2024},
  url = {https://openreview.net/forum?id=pRd1l91yMn},
}
Figure