Introduction
Video Object Detection (VOD) is a critical component for autonomous systems, yet standard RGB-based methods frequently falter in real-world "adverse" conditions. Factors such as pitch-black nights, heavy fog, or blinding glare from headlights can render visual sensors ineffective. Thermal imaging offers a solution by capturing heat signatures, which are invariant to lighting conditions. This paper introduces the Erasure-based Interaction Network (EINet) and the VT-VOD50 benchmark to advance the field of RGBT (RGB-Thermal) Video Object Detection.
The Challenge of Multi-Modal Fusion
While thermal cameras excel at capturing object contours, simply "stacking" or concatenating RGB and thermal data often introduces noise. If one modality is degraded (e.g., RGB glare), it can negatively impact the joint feature representation. Furthermore, traditional VOD methods often rely on large memory banks of previous frames to maintain temporal consistency, leading to significant computational overhead that hinders real-time performance.
The EINet Architecture
The authors propose EINet, a novel framework designed for efficient and effective RGBT fusion. It introduces two key innovations:
1. Erasure-based Interaction Module
Instead of a traditional additive fusion, EINet employs a "subtractive" approach. By utilizing a custom Negative SiLU activation function, the network identifies regions in the thermal map that correspond to background or noise. It then uses this information to "erase" the corresponding noisy features in the RGB map. This ensures that the final fused features are cleaner and more focused on the actual targets, leveraging the thermal modality as a high-quality spatial filter.
2. Temporal Proximity Enhancement (TPE)
To address the efficiency problem, the authors introduce TPE. Rather than maintaining a long-term memory bank, TPE focuses on a narrow local window—the current frame plus its immediate neighbors. By calculating spatial and channel similarities to align these frames, TPE provides temporal consistency at a fraction of the cost of long-term methods, enabling high-speed inference.
The VT-VOD50 Benchmark
A major contribution of this work is VT-VOD50, the first comprehensive benchmark dedicated to RGBT VOD. The dataset consists of 50 pairs of perfectly aligned RGBT video sequences, totaling 9,449 image pairs. It covers seven real-world traffic categories (car, van, person, etc.) across diverse conditions, including varied weather and lighting (day, dusk, night).
Key Results and Insights
EINet demonstrates a powerful balance of accuracy and speed:
- Performance: It achieved 46.32% AP50 on the VT-VOD50 dataset, significantly outperforming single-modality baselines.
- Efficiency: The model maintains an inference speed of 92.59 FPS, making it suitable for real-time autonomous driving applications.
- Ablation: The TPE module alone provided a 2.76% improvement over the RGB baseline, validating the effectiveness of local temporal alignment.
Qualitative analysis confirms that EINet successfully detects objects in scenarios where standard models fail, such as pedestrians hidden in deep shadows or vehicles obscured by intense motion blur.
Conclusion and Future Work
EINet and the VT-VOD50 benchmark provide a solid foundation for RGBT video object detection. While the current dataset is a milestone, the authors plan to expand it to over 500 sequences with more granular labels for extreme weather. Future work will also explore efficient ways to handle long-term occlusions that exceed the current three-frame temporal window.
Summary updated