VQCNIR: Clearer Night Image Restoration with Vector-Quantized Codebook

Wenbin Zou, Hongxia Gao, Tian Ye, Liang Chen, Weipeng Yang, Shasha Huang, Hongsheng Chen, Sixiang Chen

Abstract

Night photography is frequently compromised by the simultaneous presence of low-light conditions and motion blur. VQCNIR (Vector-Quantized Codebook for Night Image Restoration) addresses these challenges by utilizing a pre-trained VQ codebook as a high-quality image prior. Unlike traditional end-to-end models that learn from scratch, VQCNIR treats restoration as a matching and fusion task. It incorporates an Adaptive Illumination Enhancement Module (AIEM) to align input lighting with the codebook and Deformable Bi-directional Cross-Attention (DBCA) to correct geometric distortions. Evaluations on the LOL-Blur dataset demonstrate that VQCNIR significantly outperforms current state-of-the-art methods in both quantitative accuracy and perceptual quality.

Introduction

Capturing clear images in low-light environments remains a significant hurdle for modern imaging systems. To compensate for insufficient light, long exposure times are often employed, which introduces motion blur from camera shake or moving subjects. Existing deep learning approaches often struggle with this "double trouble," either failing to generalize to complex night scenes or producing unnatural artifacts due to a lack of robust image priors. VQCNIR proposes a data-driven solution that leverages a "memory bank" of high-quality features to guide the restoration process, ensuring results are both sharp and naturally lit.

Methodology

The VQCNIR framework is built around a Vector-Quantized (VQ) codebook derived from a VQGAN, serving as a library of sharp texture patches. The architecture consists of several specialized components designed to bridge the gap between degraded inputs and high-quality priors:

  1. Adaptive Illumination Enhancement Module (AIEM): This module uses Illumination Mutual Attention Convolution (IMAConv) to adjust the lighting of input features, ensuring they are compatible with the high-quality entries in the codebook environment.
  2. Deformable Bi-directional Cross-Attention (DBCA): To handle spatial distortions from motion blur, DBCA employs deformable convolutions to precisely align codebook features with the input image's distorted structures.
  3. Parallel Decoders: The system utilizes a fixed pre-trained decoder (Decoder G) for high-quality anchor features and a primary restorer (Decoder D) to fuse input data with the learned priors.
The VQCNIR architecture overview.

Results and Discussion

VQCNIR was rigorously tested on the LOL-Blur dataset, which includes 10,200 synthetic training pairs and a Real-LOL-Blur collection for testing generalization. Quantitatively, it achieved a PSNR of 27.79 dB, a 2.05 dB improvement over the previous state-of-the-art, LEDNet. Perceptually, its LPIPS score of 0.096 represents a nearly 2.3x improvement in visual fidelity compared to LEDNet (0.224).

Performance comparison across multiple metrics.

On real-world data, VQCNIR consistently produced the highest scores in non-reference metrics such as MUSIQ (51.04) and NRQM (7.064), confirming its ability to generate visually pleasing and sharp results in actual night scenes. The model avoids the common "hallucination" problem of AI by using bi-directional cross-attention to ensure added details remain faithful to the original scene.

Conclusion

By combining learned image priors with adaptive alignment modules, VQCNIR sets a new standard for night image restoration. While the model shows immense promise, future work will address potential structural hallucinations in cases of extreme blur and optimize the computational efficiency of the cross-attention mechanisms for real-time mobile deployment.

Summary updated

Abstract

Night photography often struggles with challenges like low light and blurring, stemming from dark environments and prolonged exposures. Current methods either disregard priors and directly fitting end-to-end networks, leading to inconsistent illumination, or rely on unreliable handcrafted priors to constrain the network, thereby bringing the greater error to the final result. We believe in the strength of data-driven high-quality priors and strive to offer a reliable and consistent prior, circumventing the restrictions of manual priors. In this paper, we propose Clearer Night Image Restoration with Vector-Quantized Codebook (VQCNIR) to achieve remarkable and consistent restoration outcomes on real-world and synthetic benchmarks. To ensure the faithful restoration of details and illumination, we propose the incorporation of two essential modules: the Adaptive Illumination Enhancement Module (AIEM) and the Deformable Bi-directional Cross-Attention (DBCA) module. The AIEM leverages the inter-channel correlation of features to dynamically maintain illumination consistency between degraded features and high-quality codebook features. Meanwhile, the DBCA module effectively integrates texture and structural information through bi-directional cross-attention and deformable convolution, resulting in enhanced fine-grained detail and structural fidelity across parallel decoders. Extensive experiments validate the remarkable benefits of VQCNIR in enhancing image quality under low-light conditions, showcasing its state-of-the-art performance on both synthetic and real-world datasets. The code is available at https://github.com/AlexZou14/VQCNIR.
BibTeX
@inproceedings{zou2024vqcnir,
  author = {Wenbin Zou and Hongxia Gao and Tian Ye and Liang Chen and Weipeng Yang and Shasha Huang and Hongsheng Chen and Sixiang Chen},
  title = {{VQCNIR: Clearer Night Image Restoration with Vector-Quantized Codebook}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28623},
}
Figure