Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection

Manish Sharma, Moitreya Chatterjee, Kuan-Chuan Peng, Suhas Lohit, Michael J. Jones

Introduction

Deep learning models typically require vast amounts of labeled data to achieve high performance. While RGB datasets for tasks like autonomous driving are abundant, labeled thermal or infrared (IR) data remains scarce and expensive to produce. Infrared imaging is vital for safety-critical applications, as it can detect heat signatures in conditions where visible-light cameras fail, such as total darkness, fog, or glare.

The primary challenge in IR object detection is data paucity. Training large-scale models like YOLOv7 on limited IR datasets often leads to overfitting. Standard domain adaptation techniques frequently struggle with the significant "modality shift" between RGB (texture/color-based) and IR (heat/intensity-based) domains.

Methodology: The TensorFact Framework

To address these challenges, researchers from the Rochester Institute of Technology and Mitsubishi Electric Research Laboratories (MERL) proposed TensorFact. This framework reimagines the weight structure of Convolutional Neural Networks (CNNs) through tensor decomposition.

Tensor decomposition of a convolutional kernel into low-rank factor matrices A and B.

The core components of the method include:

  1. Tensor Decomposition: Standard 4D convolution kernels are decomposed into two low-rank factor matrices, $A$ and $B$. This structural constraint reduces the total number of parameters, acting as a regularizer that prevents overfitting on small datasets.
  2. Cross-Modal Pre-training: These factor matrices are initially trained on large-scale RGB datasets to learn generic object features (e.g., silhouettes and shapes) common to both modalities.
  3. Capacity Augmentation: During adaptation to the IR domain, the pre-trained RGB weights are frozen. A small "parallel branch" of trainable parameters ($\Delta A$ and $\Delta B$) is added to capture IR-specific features.
  4. Complementarity Loss: A specific loss function ($L_c$) encourages the new IR-specific parameters to capture information that is distinct from the frozen RGB features, ensuring efficient knowledge transfer.

Experimental Setup

The study utilized the FLIR Aligned RGB dataset for pre-training and the FLIR ADAS v1 (Infrared) dataset as the target domain. To simulate extreme data scarcity, the authors conducted experiments using only 1% of the IR training data (approximately 62 images).

Key Results and Discussion

TensorFact demonstrates significant improvements over standard fine-tuning approaches in data-constrained scenarios.

Qualitative comparison between the baseline YOLOv7 and the proposed TensorFact method on infrared imagery.
  • Performance: On the FLIR ADAS v1 IR dataset, TensorFact achieved 62.05% mAP@50, a 4% improvement over the YOLOv7 baseline (58.49%).
  • Efficiency: The method required 95% fewer trainable parameters during the IR adaptation phase compared to the baseline.
  • Qualitative Improvements: As shown in the visual comparisons, TensorFact is notably more effective at detecting small, distant objects and those with low thermal contrast against the background.

Key Insights

  • Low-Rank Regularization: Unlike traditional uses of tensor decomposition for model compression, here it serves as a structural constraint to prevent overfitting in low-data regimes.
  • Residual Knowledge: The approach treats IR-specific features as a "residual" added to a base of universal object knowledge derived from RGB data.

Limitations and Future Directions

While effective, the current implementation is tied to the YOLOv7 architecture. Future research could explore the integration of TensorFact into transformer-based backbones (e.g., ViTs). Additionally, while the complementarity loss provides a slight boost, the architectural decomposition appears to be the primary driver of performance. Further studies could also test the method's efficacy across more diverse modalities, such as LiDAR or Synthetic Aperture Radar (SAR).

Summary updated

Abstract

The primary bottleneck towards obtaining good recognition performance in IR images is the lack of sufficient labeled training data, owing to the cost of acquiring such data. Realizing that object detection methods for the RGB modality are quite robust (at least for some commonplace classes, like person, car, etc.), thanks to the giant training sets that exist, in this work we seek to leverage cues from the RGB modality to scale object detectors to the IR modality, while preserving model performance in the RGB modality. At the core of our method, is a novel tensor decomposition method called TensorFact which splits the convolution kernels of a layer of a Convolutional Neural Network (CNN) into low-rank factor matrices, with fewer parameters than the original CNN. We first pretrain these factor matrices on the RGB modality, for which plenty of training data are assumed to exist and then augment only a few trainable parameters for training on the IR modality to avoid over-fitting, while encouraging them to capture complementary cues from those trained only on the RGB modality. We validate our approach empirically by first assessing how well our TensorFact decomposed network performs at the task of detecting objects in RGB images vis-a-vis the original network and then look at how well it adapts to IR images of the FLIR ADAS v1 dataset. For the latter, we train models under scenarios that pose challenges stemming from data paucity. From the experiments, we observe that: (i) TensorFact shows performance gains on RGB images; (ii) further, this pre-trained model, when fine-tuned, outperforms a standard state-of-the-art object detector on the FLIR ADAS v1 dataset by about 4% in terms of mAP 50 score.
BibTeX
@inproceedings{sharma2023tensor,
  author = {Manish Sharma and Moitreya Chatterjee and Kuan-Chuan Peng and Suhas Lohit and Michael J. Jones},
  title = {{Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection}},
  booktitle = {Workshops @ ICCV},
  year = {2023},
  doi = {10.1109/iccvw60793.2023.00099},
}
Figure