VIIS: Visible and Infrared Information Synthesis for Severe Low-light Image Enhancement

Chen Zhao, Mengyuan Yu, Fan Yang, Peiguang Jing

Abstract

Images captured in severe low-light circumstances often suffer from significant information absence. Existing singular modality image enhancement methods struggle to restore image regions lacking valid information. By leveraging light-impervious infrared images, visible and infrared image fusion methods have the potential to reveal information hidden in darkness. However, they primarily emphasize inter-modal complementation but neglect intra-modal enhancement, limiting the perceptual quality of output images. To address these limitations, we propose a novel task, dubbed visible and infrared information synthesis (VIIS), which aims to achieve both information enhancement and fusion of the two modalities. Given the difficulty in obtaining ground truth in the VIIS task, we design an information synthesis pretext task (ISPT) based on image augmentation. We employ a diffusion model as the framework and design a sparse attention-based dual-modalities residual (SADMR) conditioning mechanism to enhance information interaction between the two modalities. This mechanism enables features with prior knowledge from both modalities to adaptively and iteratively attend to each modality's information during the denoising process. Our extensive experiments demonstrate that our model qualitatively and quantitatively outperforms not only the state-of-the-art methods in relevant fields but also the newly designed baselines capable of both information enhancement and fusion.

Introduction

Capturing high-quality images in severe low-light environments—such as nighttime surveillance, subterranean exploration, or wildlife photography—remains a formidable challenge. In these "extreme" scenarios, the visible light signal is often so weak that it falls below the sensor's noise floor, resulting in images that are essentially devoid of useful information. Traditional low-light image enhancement (LLIE) techniques, which typically rely on boosting existing signals, struggle when the underlying information is missing.

Infrared (IR) imaging provides a robust alternative as it is "light-impervious," capturing thermal signatures regardless of ambient lighting. However, IR images lack the rich color and fine texture of visible light images, often appearing as low-resolution grayscale maps. While Visible-Infrared Fusion (VIF) methods attempt to merge these modalities, they often prioritize simple pixel-level fusion over active image enhancement. Consequently, the fused outputs frequently remain dark and perceptually unappealing. To bridge this gap, we introduce the task of Visible and Infrared Information Synthesis (VIIS), which seeks to simultaneously enhance the visible modality and fuse it with infrared data to reconstruct high-fidelity, colorized scenes.

Proposed Method: The VIIS Framework

The VIIS framework is built upon a Latent Diffusion Model (LDM), selected for its superior ability to generate high-frequency details and realistic textures. The architecture consists of two primary innovations: the SADMR conditioning mechanism and the ISPT training strategy.

The overall architecture of the VIIS framework, featuring the Multi-scale Encoder and the SADMR-guided Denoising Process.

Sparse Attention-based Dual-Modalities Residual (SADMR)

Unlike conventional fusion methods that simply concatenate multi-modal features, our approach employs a sophisticated conditioning mechanism. We design Sparse Cross-Attention Modules (SCAM) that utilize deformable attention. This allows the model to adaptively and iteratively "attend" to specific, relevant regions across both visible and infrared feature maps. By focusing on local structural alignments rather than global patterns, the model effectively learns how infrared heat signatures should guide the reconstruction of color and texture in the visible domain.

Information Synthesis Pretext Task (ISPT)

A significant obstacle in training VIIS models is the lack of ground truth data; it is nearly impossible to capture perfectly aligned "severe low-light" and "high-quality daytime" image pairs of dynamic scenes. To address this, we developed the Information Synthesis Pretext Task (ISPT). We utilize high-quality daytime images and apply a series of aggressive augmentations—including extreme gamma transforms, contrast reduction, and simulated sensor noise—to create "pseudo-low-light" visible images. This allows the model to learn the synthesis process by attempting to recover the original high-quality image from the degraded input and its corresponding IR pair.

Experimental Results

We evaluated the VIIS model on two prominent multi-spectral datasets: MSRS (Multi-Spectral Road Scenarios) and KAIST-MS. Our experiments focused on challenging nighttime subsets to test the limits of the synthesis task.

Quantitative Performance

The VIIS model consistently outperformed state-of-the-art (SOTA) methods across various perceptual and information-theoretic metrics. Notably, it achieved superior scores in Entropy (EN), indicating a higher information content, and Natural Image Quality Evaluator (NIQE). On the MSRS dataset, VIIS achieved a NIQE score of 3.57, surpassing the next best competitor. In fidelity tests (reconstructing known daytime images), our model reached a PSNR of 21.88 dB, a significant improvement over existing diffusion-based and GAN-based baselines like Retinexformer.

Qualitative Comparison

Visually, the results produced by VIIS are striking. As demonstrated in Figure 3, while other methods often produce noisy, "ghostly," or overly dark images, VIIS generates outputs with natural lighting, realistic colorization, and sharp structural details.

Qualitative comparison of VIIS against various state-of-the-art enhancement and fusion methods on the MSRS dataset.

The model successfully uses infrared information as a structural "skeleton" to fill in areas that are pitch black in the original visible input, effectively "seeing" through the darkness to reconstruct the scene.

Conclusion and Future Work

The VIIS framework represents a significant advancement in multi-modal image processing, moving beyond simple fusion toward intelligent information synthesis. By leveraging the strengths of diffusion models and sparse attention, we can recover high-quality visible images even in extreme low-light conditions.

Future research will focus on narrowing the gap between simulated and real-world noise distributions in the pretext task. Additionally, we aim to improve the preservation of unique visible-light cues—such as specific road markings—that may not have a strong infrared signature, ensuring the model remains faithful to all available information.

The code for this project is available at https://github.com/Chenz418/VIIS.

Summary updated

Abstract

Images captured in severe low-light circumstances often suffer from significant information absence. Existing singular modality image enhancement methods struggle to restore image regions lacking valid information. By leveraging light-impervious infrared images, visible and infrared image fusion methods have the potential to reveal information hidden in darkness. However, they primarily emphasize inter-modal complementation but neglect intra-modal enhancement, limiting the perceptual quality of output images. To address these limitations, we propose a novel task, dubbed visible and infrared information synthesis (VIIS), which aims to achieve both information enhancement and fusion of the two modalities. Given the difficulty in obtaining ground truth in the VIIS task, we design an information synthesis pretext task (ISPT) based on image augmentation. We employ a diffusion model as the framework and design a sparse attention-based dual-modalities residual (SADMR) conditioning mechanism to enhance information interaction between the two modalities. This mechanism enables features with prior knowledge from both modalities to adaptively and iteratively attend to each modality's information during the denoising process. Our extensive experiments demonstrate that our model qualitatively and quantitatively outperforms not only the state-of-the-art methods in relevant fields but also the newly designed baselines capable of both information enhancement and fusion. The code is available at https://github.com/Chenz418/VIIS.
BibTeX
@inproceedings{zhao2025viis,
  author = {Chen Zhao and Mengyuan Yu and Fan Yang and Peiguang Jing},
  title = {{VIIS: Visible and Infrared Information Synthesis for Severe Low-light Image Enhancement}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure