Abstract
Multispectral pedestrian detection, combining RGB and Thermal Infrared (TIR) data, is essential for autonomous vehicle safety in diverse lighting conditions. However, the high computational cost of traditional "two-stream" architectures limits real-time deployment. We introduce the Adaptive Multimodal Fusion Distillation (AMFD) framework, which effectively compresses complex two-stream teachers into efficient single-stream students. By shifting distillation from final fused outputs to individual modal features and utilizing a Modal Extraction Alignment (MEA) module, AMFD bridges the capacity gap between models. Our approach achieves state-of-the-art performance, with the student model sometimes outperforming the teacher while doubling inference speed.
Introduction
Pedestrian detection is a critical component of autonomous driving systems. While RGB cameras perform well in daylight, they struggle in low-light or adverse weather conditions. Multispectral detection addresses this by incorporating Thermal Infrared (TIR) sensors. Most high-performing models utilize a two-stream architecture, maintaining separate branches for each modality. This approach, while accurate, nearly doubles computational requirements and latency.
Knowledge Distillation (KD) is typically used to compress such models. However, a significant "capacity gap" exists: small student networks often struggle to replicate the complex fusion mechanisms of large teachers. Directly distilling fused features can introduce noise, hindering the performance of the compressed model.
Methodology: Adaptive Modal Fusion Distillation (AMFD)
The AMFD framework addresses the capacity gap by rethinking the distillation process. Instead of forcing the student to mimic the teacher's final "fused" representation, AMFD employs Front-Loaded Distillation. This involves distilling the teacher’s original, raw modal features (RGB and TIR) into the student’s early fusion layer, allowing the student to learn its own optimal integration strategy.
The core of this process is the Modal Extraction Alignment (MEA) Module, which calculates the distillation loss using two primary mechanisms:
- Global Feature Extraction: Utilizes GcBlocks to ensure the student captures broad scene context similar to the teacher.
- Focal Feature Extraction: Employs ground-truth masks to focus the student's attention on pedestrians, effectively ignoring background noise. This module applies both spatial and channel-wise attention to prioritize informative regions.
The resulting student model is a "single-stream" architecture that uses image-level fusion at the input, requiring only one backbone (e.g., ResNet18) and making it highly suitable for embedded hardware.
Key Insights: Distilling Ingredients, Not the Soup
A primary insight of AMFD is that it is more effective to show a student the "raw ingredients" (individual modal features) than the final "soup" (fused features). By not imposing the teacher's specific fusion architecture, the student is liberated from constraints it cannot efficiently replicate.
Furthermore, the MEA module's focal attention prevents the student from learning "dark knowledge" from the background, which in thermal images is often just noise. As shown in the visualization above, AMFD produces much cleaner feature maps where pedestrians are distinct from background noise compared to traditional fusion distillation.
Datasets and Evaluation
The framework was evaluated on three benchmarks:
- KAIST: The standard benchmark for multispectral pedestrian detection.
- LLVIP: Focused on nighttime surveillance challenges.
- SMOD (SJTU Multispectral Object Detection): A new dataset introduced by the authors featuring 8,676 strictly aligned image pairs with high person density and detailed occlusion annotations.
Results and Discussion
AMFD demonstrates that strategic distillation can lead to models that are both faster and more accurate.
- Efficiency: The AMFD-distilled student (Faster-RCNN with ResNet18) achieves an inference time of 0.05s, compared to the teacher's 0.12s.
- Accuracy: On the KAIST dataset, the student achieved a log-average Miss Rate (MR) of 7.23%, outperforming the teacher's 7.66%.
- Robustness: Significant improvements were noted across all datasets, including a 2.7% mAP boost on LLVIP and a 3.5% boost on the new SMOD dataset.
Conclusion
The AMFD framework successfully bridges the capacity gap in multispectral pedestrian detection distillation. By focusing on modal feature alignment and focal attention, it enables the creation of lightweight, single-stream models that maintain or exceed the accuracy of their complex two-stream counterparts. Future work will explore self-supervised attention mechanisms to reduce reliance on ground-truth masks during training.
Summary updated