VQCNIR: Clearer Night Image Restoration with Vector-Quantized Codebook

Wenbin Zou, Hongxia Gao, Tian Ye, Liang Chen, Weipeng Yang, Shasha Huang, Hongsheng Chen, Sixiang Chen

Abstract

Night photography often struggles with challenges like low light and blurring, stemming from dark environments and prolonged exposures. Current methods either disregard priors and directly fit end-to-end networks, leading to inconsistent illumination, or rely on unreliable handcrafted priors to constrain the network, thereby bringing greater error to the final result. We propose Clearer Night Image Restoration with Vector-Quantized Codebook (VQCNIR) to achieve remarkable and consistent restoration outcomes on real-world and synthetic benchmarks. To ensure the faithful restoration of details and illumination, we incorporate two essential modules: the Adaptive Illumination Enhancement Module (AIEM) and the Deformable Bi-directional Cross-Attention (DBCA) module. Extensive experiments validate the remarkable benefits of VQCNIR in enhancing image quality under low-light conditions, showcasing its state-of-the-art performance on both synthetic and real-world datasets.

Introduction

Capturing clear images at night presents a significant technical challenge. To gather sufficient light in dark environments, cameras require long exposure times, which inherently increases the risk of motion blur from camera shake or moving subjects. Restoring these images requires solving two distinct problems simultaneously: low-light enhancement and deblurring.

Existing solutions typically fall into two categories: end-to-end networks that often struggle with inconsistent lighting, or methods relying on handcrafted mathematical priors that frequently fail in complex real-world scenes. VQCNIR addresses these limitations by leveraging data-driven high-quality priors from a pre-trained Vector-Quantized (VQ) codebook, effectively "hallucinating" sharp textures while maintaining structural integrity.

Methodology

The core of VQCNIR is the use of a Vector-Quantized (VQ) Codebook, which acts as a high-resolution dictionary of image patches. The network encodes a blurry, dark image and identifies the closest matching "clean" patches in this dictionary to guide the reconstruction process.

The VQCNIR framework architecture, showing the encoder, AIEM, Vector Quantization, and parallel decoders.

To bridge the gap between degraded input features and high-quality codebook features, the framework introduces two novel components:

  1. Adaptive Illumination Enhancement Module (AIEM): This module adjusts the illumination of input features before they are matched against the codebook. By using inter-channel correlations to dynamically estimate enhancement curves, it ensures the codebook query is performed within the correct brightness range.
  2. Deformable Bi-directional Cross-Attention (DBCA): To address spatial misalignment caused by blur, DBCA utilizes a parallel decoder structure. One decoder processes high-quality codebook features, while the other focuses on the original input's structure. DBCA employs deformable convolutions to align these features, fusing sharp textures from the codebook into the structural framework of the original image.

Results and Discussion

VQCNIR was primarily evaluated on the LOL-Blur dataset, which contains over 10,000 image pairs of low-light, blurry images. The model sets a new benchmark for night image restoration across several metrics:

  • Quantitative Performance: On the synthetic LOL-Blur benchmark, VQCNIR achieved a PSNR of 27.79 dB, outperforming the previous state-of-the-art (LEDNet) by a significant 2.05 dB margin.
  • Perceptual Quality: In real-world testing, the model excelled in non-reference metrics such as MUSIQ and NIQE, which align more closely with human perception of sharpness and naturalness than standard pixel-wise comparisons.
  • Visual Fidelity: Qualitatively, the model avoids common artifacts like "ghosting" or over-smoothing, producing images that appear as though they were captured with professional lighting and stabilization.
Quantitative comparison showing VQCNIR's superior performance in PSNR vs LPIPS and other perceptual metrics.

Key insights from the study suggest that learned data-driven priors are superior to rigid handcrafted ones. Furthermore, aligning illumination is a critical prerequisite for effective codebook matching, and deformable convolutions are essential for handling the spatial smearing inherent in blurry images.

Conclusion

VQCNIR demonstrates that high-quality, data-driven priors can significantly improve the restoration of night images. While the model is highly effective, performance can degrade under extreme conditions of noise or total darkness, where codebook matching may lead to hallucinated artifacts. Future research will focus on increasing robustness to extreme noise and optimizing the architecture for real-time performance on mobile devices.

Summary updated

Abstract

Night photography often struggles with challenges like low light and blurring, stemming from dark environments and prolonged exposures. Current methods either disregard priors and directly fitting end-to-end networks, leading to inconsistent illumination, or rely on unreliable handcrafted priors to constrain the network, thereby bringing the greater error to the final result. We believe in the strength of data-driven high-quality priors and strive to offer a reliable and consistent prior, circumventing the restrictions of manual priors. In this paper, we propose Clearer Night Image Restoration with Vector-Quantized Codebook (VQCNIR) to achieve remarkable and consistent restoration outcomes on real-world and synthetic benchmarks. To ensure the faithful restoration of details and illumination, we propose the incorporation of two essential modules: the Adaptive Illumination Enhancement Module (AIEM) and the Deformable Bi-directional Cross-Attention (DBCA) module. The AIEM leverages the inter-channel correlation of features to dynamically maintain illumination consistency between degraded features and high-quality codebook features. Meanwhile, the DBCA module effectively integrates texture and structural information through bi-directional cross-attention and deformable convolution, resulting in enhanced fine-grained detail and structural fidelity across parallel decoders. Extensive experiments validate the remarkable benefits of VQCNIR in enhancing image quality under low-light conditions, showcasing its state-of-the-art performance on both synthetic and real-world datasets. The code is available at https://github.com/AlexZou14/VQCNIR.
BibTeX
@inproceedings{zou2024vqcnir,
  author = {Wenbin Zou and Hongxia Gao and Tian Ye and Liang Chen and Weipeng Yang and Shasha Huang and Hongsheng Chen and Sixiang Chen},
  title = {{VQCNIR: Clearer Night Image Restoration with Vector-Quantized Codebook}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28623},
}
Figure