TL;DR
This paper introduces Guided Attentive Feature Fusion (GAFF), a novel deep learning approach for multispectral pedestrian detection. GAFF dynamically fuses information from thermal and visible cameras using intra- and inter-modality attention mechanisms, guided by predicted pedestrian masks, to significantly improve detection accuracy with low computational cost.
What Problem Does This Paper Address?
Detecting pedestrians in real-world scenarios is challenging due to varying conditions like darkness, rain, and fog. Standard RGB cameras struggle in these situations. Multispectral systems, combining RGB and thermal infrared (IR) data, offer a promising solution by leveraging complementary information. However, effectively fusing these modalities is difficult. Existing methods often rely on hand-crafted fusion strategies or simply concatenate features, which may not adapt well to diverse conditions. This paper addresses the need for an adaptive and robust multispectral feature fusion method that can intelligently weigh and combine information from different sensors.
Methods
The core of GAFF is its novel attentive feature fusion module, integrated into a two-stream convolutional neural network architecture. GAFF employs two key attention mechanisms:
- Intra-modality Attention: This module enhances features within a single modality (thermal or visible) by highlighting areas likely to contain pedestrians. It uses predicted pedestrian masks (derived from semantic segmentation) to selectively boost relevant features, adding them as a residual to the original features to avoid drastic changes.
- Inter-modality Attention: This module adaptively selects features from the most reliable modality (thermal or visible) based on their perceived quality. It predicts attention masks that assign weights to each modality at each spatial location, ensuring their sum is one. This allows the network to dynamically favor thermal data in low-light conditions and visible data in well-lit environments.
A crucial aspect of GAFF is its guidance mechanism. Instead of solely relying on the object detection loss, GAFF introduces specific losses for the intra-modality attention (pedestrian segmentation loss) and inter-modality attention (modality selection loss). This guidance helps the fusion module learn more effectively by explicitly optimizing for accurate pedestrian masks and informed modality selection. The final fused features are a combination of features enhanced by both attention modules.
Datasets
The paper evaluates GAFF on two widely used multispectral pedestrian detection datasets:
- KAIST Multispectral Pedestrian Detection Dataset: This dataset contains approximately 95,000 aligned thermal-visible image pairs captured during day and night. The authors use both the original and a "sanitized" version of annotations, noting that the sanitized version significantly improves detection accuracy.
- FLIR ADAS Dataset: This dataset includes 5,142 aligned multispectral image pairs, covering "person," "car," and "bicycle" categories.
Key Insights and Ideas
- Adaptive Fusion is Key: Simply concatenating or adding features from different modalities is suboptimal. The system needs to intelligently adapt its fusion strategy based on the input data.
- Attention as a Guiding Force: Both intra- and inter-modality attention mechanisms are crucial. Intra-modality attention helps focus on pedestrian regions within each sensor's view, while inter-modality attention helps select the best sensor's information.
- Explicit Guidance Improves Fusion: Training the fusion module solely with the object detection loss is insufficient. Introducing specific losses for attention mask prediction (segmentation and modality selection) significantly boosts performance. This "guided" approach is a core innovation.
- Thermal vs. Visible Strengths: The experiments consistently show that thermal images are generally better for recognition, aligning with the intuition that they are less affected by lighting conditions. However, visible features are still valuable in well-lit scenarios.
- "Cold Start" Problem Mitigation: The inter-modality attention mechanism's training is designed to avoid a "cold start" problem by making modality selection less critical when initial pedestrian masks are poor.
Key Results
- State-of-the-Art Performance: GAFF achieves state-of-the-art results on the KAIST dataset, outperforming previous methods in terms of Miss Rate, especially with "sanitized" annotations. For instance, with the VGG16 backbone and sanitized annotations, GAFF achieves a Miss Rate of 6.48% (All), significantly improving upon previous methods.
- Improved Accuracy on FLIR: On the FLIR ADAS dataset, GAFF provides an average precision improvement of around 1% mAP across IoU thresholds, demonstrating its generalization capabilities.
- Low Computational Cost: GAFF adds minimal parameters and inference time (less than 0.1% additional parameters and ~0.5ms inference time on a 1080Ti), making it suitable for real-time applications and embedded devices.
- Effectiveness of Guidance: The paper shows that the guided approach (with specific fusion losses) significantly outperforms a non-guided attentive fusion method, improving the Miss Rate from 8.38% to 6.48% on the KAIST dataset.
Limitations and Future Work
- Misalignment Sensitivity: The authors note that thermal-visible image pairs in the datasets can be slightly misaligned, which could be a critical issue in real-world applications.
- Error Case Analysis: While GAFF is generally robust, an interesting error case is highlighted where a pedestrian is missed due to being absent in the thermal image, even though the guided attention mechanism selected thermal features for a uniform area. This suggests that while GAFF is adaptive, it's not infallible.
- Future Work: The authors propose developing a real-time feature calibration module based on the predicted attention masks from GAFF. This aims to address the misalignment issue and further enhance robustness.
Summary updated