Low-Light Image Enhancement with Wavelet-Based Diffusion Models

Hai Jiang, Ao Luo, Haoqiang Fan, Songchen Han, Shuaicheng Liu

TL;DR

This paper introduces DiffLL, a novel approach for low-light image enhancement that combines the power of diffusion models with wavelet transformations. By performing diffusion in the wavelet domain, DiffLL significantly accelerates inference and reduces computational costs while achieving state-of-the-art results in image quality and detail preservation.

What Problem Does This Paper Address?

Capturing clear and detailed images in low-light conditions is a persistent challenge. Existing methods often struggle with issues like noise, poor contrast, color distortion, and artifacts, hindering their performance in downstream tasks like object detection and image analysis. While diffusion models have shown promise in image restoration, they are notoriously slow and computationally expensive. This paper aims to bridge this gap by developing a diffusion-based method that is both effective and efficient for low-light image enhancement.

Methods

The core of DiffLL is a Wavelet-based Conditional Diffusion Model (WCDM). Instead of applying diffusion directly to the image pixels, DiffLL first transforms the low-light image into the wavelet domain using K-times 2D Discrete Wavelet Transforms (2D-DWT). This process decomposes the image into an average coefficient (representing global information) and several sets of high-frequency coefficients (representing local details like edges and textures).

The diffusion process is then applied only to the average coefficient. This significantly reduces the spatial dimensions, leading to faster inference and lower computational demands. To further enhance efficiency and stability, DiffLL incorporates a novel training strategy where both forward diffusion and denoising occur during training. This helps the model learn consistent denoising and avoid the chaotic content sometimes seen in diffusion models.

Additionally, a High-Frequency Restoration Module (HFRM) is introduced to reconstruct the high-frequency wavelet coefficients. This module leverages cross-attention mechanisms and progressive dilation to effectively combine vertical, horizontal, and diagonal details, ensuring fine-grained restoration.

Datasets

The paper uses several datasets for training and evaluation:

  • LOLv1: A widely used dataset for low-light image enhancement, containing both synthetic and real-world paired images.
  • LOLv2-real: Another real-world paired dataset for low-light enhancement.
  • LSRW: A dataset with 5,650 image pairs captured in various scenes.
  • UHD-LL: A dataset specifically designed for high-resolution (4K) low-light image enhancement.
  • DICM, LIME, NPE, MEF, VV: Unpaired datasets used for evaluating generalization capabilities.
  • DARK FACE: A dataset used to evaluate the impact of LLIE on face detection performance.

Key Insights and Ideas

  • Wavelet Domain Diffusion is Key: Applying diffusion in the wavelet domain, specifically on the average coefficient, is a crucial insight. It drastically reduces the computational burden without sacrificing essential image information, unlike VAE-based latent space diffusion which can lead to information loss.
  • Training Strategy for Consistency: Performing denoising during the training phase, in addition to forward diffusion, is vital for achieving content consistency during inference. This mitigates the randomness inherent in diffusion models and leads to more stable and predictable results.
  • High-Frequency Details Matter: Simply enhancing the global illumination isn't enough. The HFRM's ability to effectively reconstruct high-frequency details by intelligently combining different wavelet sub-bands is critical for producing visually rich and sharp images.
  • Efficiency Gains are Substantial: The combination of wavelet transformation and the optimized training strategy leads to significant speedups (over 70x faster than previous diffusion methods) and reduced memory usage, making diffusion models practical for LLIE.

Key Results

  • State-of-the-Art Performance: DiffLL achieves state-of-the-art results on multiple benchmark datasets (LOLv1, LOLv2-real, LSRW, UHD-LL) in terms of both distortion metrics (PSNR, SSIM) and perceptual metrics (LPIPS, FID).
  • Significant Speedup: The method is at least 70x faster than previous diffusion-based methods, with inference times for a 600x400 image being around 0.157s.
  • Reduced Computational Cost: DiffLL requires significantly less GPU memory, making it more accessible and capable of handling higher resolutions. For example, it can process 2K images without running out of memory, unlike many other methods.
  • Improved Face Detection: When used as a pre-processing step for face detection on the DARK FACE dataset, DiffLL improves the Average Precision (AP) from 26.4% to 38.5%.
  • Qualitative Superiority: Visual comparisons show DiffLL producing images with better contrast, sharper details, and fewer artifacts compared to existing methods.

Limitations and Future Work

  • Extreme Low-Light Conditions: The paper acknowledges that DiffLL's performance may degrade in extremely low-light scenarios where information loss is severe.
  • Real-time Video: The current implementation is not yet efficient enough for real-time video enhancement.
  • Task Specificity: The model is currently optimized for LLIE and its generalization to other image restoration tasks needs further investigation.

Future work could focus on extending the method to handle more extreme low-light conditions and exploring its application to other image restoration tasks.

Summary updated

BibTeX
@article{jiang2023low,
  author = {Hai Jiang and Ao Luo and Haoqiang Fan and Songchen Han and Shuaicheng Liu},
  title = {{Low-Light Image Enhancement with Wavelet-Based Diffusion Models}},
  journal = {ACM Trans. Graphics},
  year = {2023},
  doi = {10.1145/3618373},
}

Loading related content...

Figure