VIIS: Visible and Infrared Information Synthesis for Severe Low-Light Image Enhancement

Chen Zhao, Mengyuan Yu, Fan Yang, Peiguang Jing

TL;DR

This paper introduces Visible and Infrared Information Synthesis (VIIS), a novel task for enhancing severely low-light visible images by leveraging infrared imagery. They propose a diffusion model-based approach with a novel Sparse Attention-based Dual-modalities Residual (SADMR) conditioning mechanism to effectively fuse and enhance information from both modalities, outperforming existing methods in qualitative and quantitative evaluations.

What Problem Does This Paper Address?

Capturing images in extremely low-light conditions poses a significant challenge for traditional image enhancement techniques. Visible light images often suffer from severe information loss, making it impossible to recover details solely from the visible spectrum. While infrared images are unaffected by low light, they lack color and fine texture information. Existing methods for visible-infrared fusion (VIF) primarily focus on merging complementary information but neglect to enhance the individual modalities, leading to fused images that still exhibit low contrast in visible regions and a sketchy appearance from infrared data. This paper addresses the need for a method that can simultaneously enhance the quality of visible images and effectively integrate information from infrared images to produce perceptually superior results in challenging low-light scenarios.

Methods

The authors propose a novel task called Visible and Infrared Information Synthesis (VIIS) and tackle it using a diffusion model (DM) framework. Since obtaining ground truth for VIIS is difficult, they introduce an Information Synthesis Pretext Task (ISPT). This ISPT uses data augmentation to create pseudo-low-light visible images from high-quality daytime images, which the model then learns to enhance and fuse with corresponding infrared images.

A core innovation is the Sparse Attention-based Dual-modalities Residual (SADMR) conditioning mechanism. This mechanism is designed to facilitate effective interaction between visible and infrared image features within the diffusion model's denoising network. It allows intermediate features, which have learned from both modalities, to adaptively attend to features from each modality. Specifically, it employs a Sparse Cross-Attention Module (SCAM), which uses deformable attention to enable features to selectively attend to surrounding elements in both infrared and visible image feature maps. This allows for a more nuanced integration of information compared to simple concatenation or addition.

Datasets

The paper utilizes two publicly available datasets for training and evaluation:

  • MSRS [47]: Comprises aligned visible and infrared image pairs.
  • KAIST-MS [17]: Also contains aligned visible and infrared image pairs.

For training, the authors use the daytime subsets of MSRS and KAIST-MS. For testing, they specifically select image pairs from the testing sets where visible images are severely degraded by realistic low-light conditions.

Key Insights and Ideas

  • VIIS as a Novel Task: The paper establishes VIIS as a distinct and necessary task, moving beyond simple enhancement or fusion by aiming for simultaneous dual-modal enhancement and integration.
  • ISPT for Supervision: The creation of an information synthesis pretext task is a clever way to generate supervision for a task where ground truth is inherently difficult to obtain.
  • SADMR for Cross-Modal Interaction: The SADMR mechanism, particularly the SCAM, is crucial for enabling the diffusion model to effectively leverage information from both visible and infrared modalities, leading to better feature integration.
  • Diffusion Models for Complex Fusion: The adoption of diffusion models, known for their generative capabilities, proves effective for the complex task of synthesizing information from multiple modalities under challenging conditions.

Key Results

The proposed VIIS method, implemented with the diffusion model and SADMR mechanism, demonstrates superior performance compared to state-of-the-art methods in low-light image enhancement, infrared image colorization, and visible-infrared image fusion.

  • Qualitative Results: Visual comparisons on MSRS and KAIST-MS datasets show that the VIIS model effectively restores severely corrupted regions, maintains authentic colors, and produces images with higher perceptual quality than existing methods. Single-modality enhancement methods fail to recover lost information, while VIF methods often result in sketchy or low-contrast outputs.
  • Quantitative Results: The model achieves the best performance across several metrics, including EN (entropy), NIQE, and BRISQUE, on both MSRS and KAIST-MS datasets. While it achieves the second-best result for the SD (standard deviation) metric on KAIST-MS (with MUGAN achieving the best), its overall performance is consistently strong.
  • Ablation Studies: Experiments confirm the effectiveness of the SADMR conditioning mechanism and the SCAM module, showing that their removal or replacement leads to a decline in performance. The study also highlights the necessity of both visible and infrared images for optimal results.

Limitations and Future Work

  • Pretext Task Simplicity: The authors acknowledge that their ISPT, while effective, is relatively simple and may not be as sophisticated as pretext tasks in more established fields. They note that it might sometimes weaken information unique to visible images (e.g., texture or color), as illustrated in Figure 7.
  • Future Directions: The paper suggests that their ISPT provides a novel approach for guiding comprehensive information enhancement and fusion, inspiring further research in related communities. The VIIS framework has promising applications in various low-light imaging fields, including surveillance and wildlife photography.

Summary updated

Abstract

Images captured in severe low-light circumstances often suffer from significant information absence. Existing singular modality image enhancement methods struggle to restore image regions lacking valid information. By leveraging light-impervious infrared images, visible and infrared image fusion methods have the potential to reveal information hidden in darkness. However, they primarily emphasize inter-modal complementation but neglect intra-modal enhancement, limiting the perceptual quality of output images. To address these limitations, we propose a novel task, dubbed visible and infrared information synthesis (VIIS), which aims to achieve both information enhancement and fusion of the two modalities. Given the difficulty in obtaining ground truth in the VIIS task, we design an information synthesis pretext task (ISPT) based on image augmentation. We employ a diffusion model as the framework and design a sparse attention-based dual-modalities residual (SADMR) conditioning mechanism to enhance information interaction between the two modalities. This mechanism enables features with prior knowledge from both modalities to adaptively and iteratively attend to each modality's information during the denoising process. Our extensive experiments demonstrate that our model qualitatively and quantitatively outperforms not only the state-of-the-art methods in relevant fields but also the newly designed baselines capable of both information enhancement and fusion. The code is available at https://github.com/Chenz418/VIIS.
BibTeX
@inproceedings{zhao2025viis,
  author = {Chen Zhao and Mengyuan Yu and Fan Yang and Peiguang Jing},
  title = {{VIIS: Visible and Infrared Information Synthesis for Severe Low-Light Image Enhancement}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure