Abstract
Traditional visible and infrared image fusion (VIF) systems often struggle with motion blur and limited dynamic range due to the constraints of frame-based cameras. This paper introduces EVIF, a novel framework that utilizes event cameras to overcome these limitations. By employing a multi-task collaborative approach—integrating visible texture reconstruction, event-guided infrared deblurring, and feature-level fusion—EVIF produces high-quality, sharp fused images even in challenging high-speed and low-light scenarios.
Introduction
Visible and infrared (thermal) image fusion is critical for applications like autonomous driving and surveillance, where combining texture and heat signatures provides a comprehensive scene understanding. However, standard frame-based cameras are prone to motion blur and overexposure in dynamic or extreme lighting conditions. Infrared sensors also exhibit blurriness during fast motion. These degradations in input quality inevitably lead to poor fusion results.
Methodology
The proposed Event-based Visible and Infrared Fusion (EVIF) system replaces traditional visible cameras with event cameras. Event cameras record per-pixel brightness changes with microsecond latency and high dynamic range (>120dB), making them inherently robust to motion blur and exposure issues.
The EVIF framework consists of three collaborative tasks:
- Event-based Visible Texture Reconstruction: Reconstructs visible textures from sparse event streams.
- Event-guided Infrared Image Deblurring: Leverages high-temporal-resolution event data to deblur infrared frames.
- Visible-Infrared Fusion: Merges the reconstructed visible features and deblurred infrared features.
A key innovation is the Cross-task Event Enhancement module, which facilitates feature sharing between reconstruction and deblurring. Furthermore, a Bi-level Min-Max Mutual Information (MI) Optimization strategy is used to minimize redundancy between modalities while maximizing information retention in the fused output.
Experimental Setup and Results
The system was evaluated on both synthetic and real-world data. Synthetic datasets were derived from the KAIST multi-spectral benchmark with simulated motion blur and events. Real-world data was collected using a custom hardware prototype featuring a DAVIS346 event camera and a synchronized infrared sensor.
EVIF demonstrates significant improvements over state-of-the-art (SOTA) frame-based fusion methods. Quantitatively, it leads in information-theoretic metrics like Entropy (EN) and Mutual Information (MI). Qualitatively, EVIF maintains sharp edges and clear outlines in high-speed captures where traditional methods produce significant ghosting and blur. The "Bi-level MI" optimization was found to be critical; without it, fused images often appeared as simple, washed-out averages of the inputs.
Conclusion
EVIF represents a significant step forward in robust image fusion by leveraging the unique properties of event cameras. By treating reconstruction, deblurring, and fusion as a unified collaborative process, the system effectively overcomes the physical limitations of traditional sensors. While challenges remain in automated hardware alignment and computational optimization for real-time edge devices, the multi-task collaborative approach proves highly effective at maintaining clarity in extreme environments.
Summary updated