TL;DR
This paper introduces Modality Distillation (MD), a novel knowledge distillation framework that enables high-performance multispectral scene analysis using low-resolution thermal cameras and simpler one-stream networks. By distilling knowledge from a complex two-stream teacher model to a more efficient student model, MD significantly reduces computational costs and hardware requirements, making advanced multispectral vision practical for embedded systems.
What Problem Does This Paper Address?
Multispectral scene analysis, which combines visible and thermal imagery, offers superior reliability for tasks like autonomous driving and surveillance, especially in challenging lighting conditions. However, its widespread adoption is hindered by two major practical limitations: 1) High-resolution thermal cameras are prohibitively expensive, and 2) common two-stream neural network architectures, which process each modality separately before fusion, double inference time, making them unsuitable for resource-constrained embedded devices. This paper aims to bridge this gap by enabling effective multispectral analysis with affordable, low-resolution thermal sensors and computationally efficient models.
Methods
The core of this work is the Modality Distillation (MD) framework, which employs a two-stage training process:
- Teacher Model Training: A high-resolution, two-stream multispectral network with feature-level fusion (specifically using the Guided Attentive Feature Fusion - GAFF - module from [27]) is trained. This model serves as the "teacher," learning rich semantic representations from high-quality data.
- Student Model Training: A more efficient one-stream network is trained. This "student" model takes a low-resolution thermal image (downsampled) and a high-resolution visible image as input. Knowledge transfer from the teacher to the student occurs through two novel distillation modules:
- Attention Transfer: This module aims to mimic the teacher's attentive fusion process. The student generates attention masks that are supervised by the teacher's attention masks (derived from GAFF), encouraging the student to focus on relevant features.
- Semantic Transfer: This module addresses the resolution degradation of the thermal input. It uses a residual block and a novel Focal Mean Square Error (F-MSE) loss to encourage the student's fused features to match the teacher's high-resolution fused features. The F-MSE loss adaptively re-weights feature differences, focusing on challenging areas to overcome imbalances between easily and hardly mimicked regions.
The student model uses an image-level fusion approach (concatenating modalities at the input layer), which is computationally much cheaper than the feature-level fusion of the teacher.
Datasets
The paper evaluates its method on two established multispectral datasets:
- KAIST Multispectral Pedestrian Detection Dataset: Used for pedestrian detection.
- MFNet Multispectral Semantic Segmentation Dataset: Used for semantic segmentation of street scenes.
The authors also mention using improved annotations for the KAIST dataset from previous works.
Key Insights and Ideas
- The "Practical Compromise": The paper argues that using high-resolution visible cameras with low-resolution thermal cameras is the most practical and cost-effective setup for real-world multispectral applications.
- Image-Level Fusion's Underappreciated Potential: While feature-level fusion has dominated research due to its accuracy, the paper revives interest in image-level fusion by showing it can achieve competitive performance when guided by a strong teacher model.
- Focal Mean Square Error (F-MSE) for Feature Mimicry: The introduction of F-MSE loss is a key innovation. It effectively tackles the problem of feature imbalance during distillation by focusing learning on difficult-to-mimic areas, leading to better reconstruction of high-resolution details.
- Thermal Modality's Importance Even at Low Resolution: The results demonstrate that even significantly downsampled thermal data, when effectively distilled, provides crucial information, especially for nighttime performance, and can even improve visible-only models.
Key Results
- Significant Performance Gains: The Modality Distillation (MD) strategy consistently improves performance across various thermal resolutions for both pedestrian detection and semantic segmentation on both datasets.
- Reduced Inference Time: The distilled one-stream student models achieve inference times that are roughly halved compared to the two-stream teacher models, while maintaining comparable accuracy. For example, on KAIST, the runtime is reduced from 10.97ms to 6.55ms.
- Low-Resolution Thermal Performance: Even with thermal resolution reduced by 16x, the MD strategy achieves only a minor accuracy drop (0.26% Miss Rate on KAIST) compared to the full-resolution teacher.
- Competitive with State-of-the-Art: The distilled "practical" student models (using 16x downsampled thermal input) achieve performance comparable to or better than existing state-of-the-art feature-level fusion methods, despite using simpler architectures and lower-resolution thermal data.
- Nighttime Boost: The method shows particularly strong improvements in nighttime detection accuracy.
- Visible-Only Improvement: Distilling multispectral knowledge even benefits a visible-only student model, suggesting a form of "pseudo-multispectral" detection.
Limitations and Future Work
- Limitations: The paper acknowledges that while the student model's performance is close to the teacher's, there is still a slight precision drop. The effectiveness of the F-MSE loss is demonstrated, but its theoretical underpinnings could be further explored.
- Future Work: The authors propose extending their method to scene analysis tasks involving even more modalities, such as depth sensors, Doppler radar, and LiDAR.
Summary updated