Event-Based Visible and Infrared Fusion via Multi-Task Collaboration

Mengyue Geng, Lin Zhu, Lizhi Wang, Wei Zhang, Ruiqin Xiong, Yonghong Tian

Abstract

Traditional visible and infrared image fusion (VIF) systems often struggle with motion blur and limited dynamic range due to the constraints of frame-based cameras. This paper introduces EVIF, a novel framework that utilizes event cameras to overcome these limitations. By employing a multi-task collaborative approach—integrating visible texture reconstruction, event-guided infrared deblurring, and feature-level fusion—EVIF produces high-quality, sharp fused images even in challenging high-speed and low-light scenarios.

Introduction

Visible and infrared (thermal) image fusion is critical for applications like autonomous driving and surveillance, where combining texture and heat signatures provides a comprehensive scene understanding. However, standard frame-based cameras are prone to motion blur and overexposure in dynamic or extreme lighting conditions. Infrared sensors also exhibit blurriness during fast motion. These degradations in input quality inevitably lead to poor fusion results.

Comparison of visible, infrared, and event camera characteristics and their fusion results.

Methodology

The proposed Event-based Visible and Infrared Fusion (EVIF) system replaces traditional visible cameras with event cameras. Event cameras record per-pixel brightness changes with microsecond latency and high dynamic range (>120dB), making them inherently robust to motion blur and exposure issues.

The EVIF framework consists of three collaborative tasks:

  1. Event-based Visible Texture Reconstruction: Reconstructs visible textures from sparse event streams.
  2. Event-guided Infrared Image Deblurring: Leverages high-temporal-resolution event data to deblur infrared frames.
  3. Visible-Infrared Fusion: Merges the reconstructed visible features and deblurred infrared features.
The EVIF framework integrates three tasks: visible texture reconstruction, infrared deblurring, and feature fusion.

A key innovation is the Cross-task Event Enhancement module, which facilitates feature sharing between reconstruction and deblurring. Furthermore, a Bi-level Min-Max Mutual Information (MI) Optimization strategy is used to minimize redundancy between modalities while maximizing information retention in the fused output.

Experimental Setup and Results

The system was evaluated on both synthetic and real-world data. Synthetic datasets were derived from the KAIST multi-spectral benchmark with simulated motion blur and events. Real-world data was collected using a custom hardware prototype featuring a DAVIS346 event camera and a synchronized infrared sensor.

EVIF demonstrates significant improvements over state-of-the-art (SOTA) frame-based fusion methods. Quantitatively, it leads in information-theoretic metrics like Entropy (EN) and Mutual Information (MI). Qualitatively, EVIF maintains sharp edges and clear outlines in high-speed captures where traditional methods produce significant ghosting and blur. The "Bi-level MI" optimization was found to be critical; without it, fused images often appeared as simple, washed-out averages of the inputs.

Conclusion

EVIF represents a significant step forward in robust image fusion by leveraging the unique properties of event cameras. By treating reconstruction, deblurring, and fusion as a unified collaborative process, the system effectively overcomes the physical limitations of traditional sensors. While challenges remain in automated hardware alignment and computational optimization for real-time edge devices, the multi-task collaborative approach proves highly effective at maintaining clarity in extreme environments.

Summary updated

Abstract

Visible and Infrared image Fusion (VIF) offers a comprehensive scene description by combining thermal infrared images with the rich textures from visible cameras. However, conventional VIF systems may capture over/under exposure or blurry images in extreme lighting and high dynamic motion scenarios, leading to degraded fusion results. To address these problems, we propose a novel Event-based Visible and Infrared Fusion (EVIF) system that employs a visible event camera as an alternative to traditional framebased cameras for the VIF task. With extremely low latency and high dynamic range, event cameras can effectively address blurriness and are robust against diverse luminous ranges. To produce high-quality fused images, we develop a multi-task collaborative framework that simultaneously performs event-based visible texture reconstruction, eventguided infrared image deblurring, and visible-infrared fusion. Rather than independently learning these tasks, our framework capitalizes on their synergy, leveraging crosstask event enhancement for efficient deblurring and bi-level min-max mutual information optimization to achieve higher fusion quality. Experiments on both synthetic and real data show that EVIF achieves remarkable performance in dealing with extreme lighting conditions and high-dynamic scenes, ensuring high-quality fused images across a broad range of practical scenarios.
BibTeX
@inproceedings{geng2024event,
  author = {Mengyue Geng and Lin Zhu and Lizhi Wang and Wei Zhang and Ruiqin Xiong and Yonghong Tian},
  title = {{Event-Based Visible and Infrared Fusion via Multi-Task Collaboration}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02543},
}
Figure