FDMLNet: A Frequency-Division and Multiscale Learning Network for Enhancing Low-Light Image

Haoxiang Lu, Junming Gong, Zhenbing Liu, Rushi Lan, Xipeng PanSensors 2022 · DOI

TL;DR

This paper introduces FDMLNet, a novel deep learning approach for enhancing low-light images. By separating images into high and low-frequency components and processing them with specialized sub-networks, FDMLNet effectively boosts brightness and contrast while preserving details and reducing noise, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Low-light images, common in scenarios like nighttime photography or underexposed shots, suffer from poor brightness, blurry details, and color casts. This not only degrades the visual experience but also significantly hinders the performance of downstream computer vision tasks like object recognition. While many low-light image enhancement (LLIE) methods exist, they often struggle with issues like noise amplification, color distortion, or introduce artificial artifacts. Existing deep learning approaches, while powerful, can be computationally expensive and may not fully leverage hierarchical and multiscale image information.

Methods

FDMLNet tackles these challenges with a multi-pronged approach:

  • Frequency Division (FD): The core idea is that different image information resides at different frequencies. High frequencies capture details, edges, and noise, while low frequencies represent structure and background. FDMLNet uses a guided filter to effectively separate these components from the input low-light image.
  • DetNet (Detail Network): This sub-network is designed to process the high-frequency components. It uses a series of convolutional layers to preserve fine details and suppress noise.
  • StruNet (Structure Network): This sub-network handles the low-frequency components, focusing on enhancing brightness and contrast. It incorporates:
  • Feasible Feature Extraction Module (FFEM): This module is the heart of StruNet. It comprises three stacked Multiscale Attention Modules (MSAMs) connected via a dense skip-connection strategy.
  • Multiscale Learning Block (MSL): Within each MSAM, this block uses parallel 3x3 convolutions with different dilation rates (1, 2, 3, and 5) to capture information at various scales. Features from earlier branches are fed into later ones to leverage multiscale context.
  • Dual-Branch Channel Attention Mechanism (DCAM): Also within each MSAM, DCAM employs both spatial and pixel attention. This allows the network to selectively focus on the most informative regions and features, adaptively re-weighting them.
  • Loss Function: A hybrid loss function is used, combining L1-norm loss (for overall brightness), SSIM loss (for structural similarity), Total Variation (TV) loss (for noise compression), and Color Constancy (CC) loss (to correct color deviation).

Datasets

The paper evaluates FDMLNet on several public benchmarks:

  • Paired Datasets:
    • LOL: Contains 500 pairs of abnormal/normal light RGB images.
  • MIT-Adobe FiveK: A larger dataset of 5000 RAW images processed by professional photographers.
  • Unpaired Datasets:
    • LIME: 10 images.
  • MEF: 17 images.
  • NPE: 84 images.
  • VV: 24 images.

The authors also tested their model on real low-light images captured by Mate 20 Pro and Vivo X60 phones, as well as compressed low-light images.

Key Insights and Ideas

  • Frequency-domain processing is crucial: The paper strongly advocates for separating image information based on frequency. This allows for specialized processing tailored to the characteristics of high-frequency (detail/noise) and low-frequency (structure/brightness) components, which is more effective than treating the image as a whole.
  • Multiscale learning is underutilized in LLIE: While common in other vision tasks, multiscale feature extraction is not widely explored in LLIE. FDMLNet's MSAM effectively integrates multiscale information, leading to richer feature representations.
  • Attention mechanisms enhance focus: The DCAM, by incorporating both spatial and pixel attention, helps the model pinpoint and leverage the most critical image features, improving the quality of enhancement.
  • Dense connectivity for feature fusion: Connecting FFEMs with dense skip-connections ensures that multilevel features are effectively assimilated, preventing information loss.

Key Results

  • Quantitative Performance: FDMLNet achieved state-of-the-art results on both paired (LOL, MIT-Adobe) and unpaired (LIME, MEF, NPE, VV) datasets. For instance, on the LOL dataset, it achieved the highest PSNR (24.658) and SSIM (0.886) among the tested methods. On the MIT-Adobe dataset, it achieved the highest PSNR (21.361) and competitive SSIM (0.879). On unpaired datasets, it achieved the lowest NIQE (2.3369) and highest PCQI (0.9801) on the VV benchmark.
  • Qualitative Performance: Visual comparisons show FDMLNet producing images with significantly improved brightness, vivid colors, and clearer details, while effectively suppressing noise and avoiding artifacts like halos or color casts, outperforming 12 other state-of-the-art methods.
  • Efficiency: FDMLNet demonstrates a good balance between performance and computational cost, with fewer parameters and faster execution times compared to many other advanced methods.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Handling Excessive Noise and Colored Light: In certain challenging scenes with very high noise levels or strong colored lighting, FDMLNet's performance degrades.
  • Hand-crafted Halos in Compressed Images: For highly compressed images (e.g., compression ratio of 0.2), the method struggled to completely remove hand-crafted halos.
  • Lack of Semantic Information: The current model does not explicitly incorporate semantic scene understanding, which could further improve enhancement.

Future work will focus on addressing these issues by:

  • Integrating semantic information into the model.
  • Building more diverse training datasets that include challenging scenes like those with high noise or colored light.
  • Potentially incorporating denoising operations directly into the DetNet.

Summary updated

BibTeX
@article{lu2022fdmlnet,
  author = {Haoxiang Lu and Junming Gong and Zhenbing Liu and Rushi Lan and Xipeng Pan},
  title = {{FDMLNet: A Frequency-Division and Multiscale Learning Network for Enhancing Low-Light Image}},
  journal = {Sensors},
  year = {2022},
  doi = {10.3390/s22218244},
}
Figure