Introduction
Deep learning models typically require vast amounts of labeled data to achieve high performance. While RGB datasets for tasks like autonomous driving are abundant, labeled thermal or infrared (IR) data remains scarce and expensive to produce. Infrared imaging is vital for safety-critical applications, as it can detect heat signatures in conditions where visible-light cameras fail, such as total darkness, fog, or glare.
The primary challenge in IR object detection is data paucity. Training large-scale models like YOLOv7 on limited IR datasets often leads to overfitting. Standard domain adaptation techniques frequently struggle with the significant "modality shift" between RGB (texture/color-based) and IR (heat/intensity-based) domains.
Methodology: The TensorFact Framework
To address these challenges, researchers from the Rochester Institute of Technology and Mitsubishi Electric Research Laboratories (MERL) proposed TensorFact. This framework reimagines the weight structure of Convolutional Neural Networks (CNNs) through tensor decomposition.
The core components of the method include:
- Tensor Decomposition: Standard 4D convolution kernels are decomposed into two low-rank factor matrices, $A$ and $B$. This structural constraint reduces the total number of parameters, acting as a regularizer that prevents overfitting on small datasets.
- Cross-Modal Pre-training: These factor matrices are initially trained on large-scale RGB datasets to learn generic object features (e.g., silhouettes and shapes) common to both modalities.
- Capacity Augmentation: During adaptation to the IR domain, the pre-trained RGB weights are frozen. A small "parallel branch" of trainable parameters ($\Delta A$ and $\Delta B$) is added to capture IR-specific features.
- Complementarity Loss: A specific loss function ($L_c$) encourages the new IR-specific parameters to capture information that is distinct from the frozen RGB features, ensuring efficient knowledge transfer.
Experimental Setup
The study utilized the FLIR Aligned RGB dataset for pre-training and the FLIR ADAS v1 (Infrared) dataset as the target domain. To simulate extreme data scarcity, the authors conducted experiments using only 1% of the IR training data (approximately 62 images).
Key Results and Discussion
TensorFact demonstrates significant improvements over standard fine-tuning approaches in data-constrained scenarios.
- Performance: On the FLIR ADAS v1 IR dataset, TensorFact achieved 62.05% mAP@50, a 4% improvement over the YOLOv7 baseline (58.49%).
- Efficiency: The method required 95% fewer trainable parameters during the IR adaptation phase compared to the baseline.
- Qualitative Improvements: As shown in the visual comparisons, TensorFact is notably more effective at detecting small, distant objects and those with low thermal contrast against the background.
Key Insights
- Low-Rank Regularization: Unlike traditional uses of tensor decomposition for model compression, here it serves as a structural constraint to prevent overfitting in low-data regimes.
- Residual Knowledge: The approach treats IR-specific features as a "residual" added to a base of universal object knowledge derived from RGB data.
Limitations and Future Directions
While effective, the current implementation is tied to the YOLOv7 architecture. Future research could explore the integration of TensorFact into transformer-based backbones (e.g., ViTs). Additionally, while the complementarity loss provides a slight boost, the architectural decomposition appears to be the primary driver of performance. Further studies could also test the method's efficacy across more diverse modalities, such as LiDAR or Synthetic Aperture Radar (SAR).
Summary updated