Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection

Manish Sharma, Moitreya Chatterjee, Kuan-Chuan Peng, Suhas Lohit, Michael Jones

TL;DR

Researchers have developed TensorFact, a novel tensor decomposition method that enables high-performance object detection in infrared (IR) imagery even when labeled data is extremely scarce. By leveraging knowledge from abundant RGB datasets and using a parameter-efficient parallel branch to capture IR-specific cues, the method outperforms state-of-the-art baselines by 4% mAP while significantly reducing the risk of overfitting.

What Problem Does This Paper Address?

Object detection in standard RGB images has reached a high level of maturity, thanks to massive labeled datasets like COCO. However, for applications like autonomous driving, RGB cameras fall short in low-light, foggy, or nighttime conditions. Thermal Infrared (IR) cameras are ideal for these scenarios because they detect heat rather than visible light.

The major hurdle is data paucity: labeling IR images is expensive and time-consuming, leading to a lack of large-scale datasets. When we try to train modern, over-parameterized deep networks (like YOLOv7) on small IR datasets, they tend to overfit, losing their ability to generalize. Furthermore, standard domain adaptation techniques often fail because the "modality shift" between RGB (color/texture) and IR (heat signatures) is too extreme for simple transfer learning.

Methods

The authors introduce TensorFact, a method that reimagines how convolutional layers learn across different modalities.

  1. Tensor Decomposition: Instead of using standard 4D convolution kernels, TensorFact decomposes each kernel into two low-rank factor matrices ($A$ and $B$). This significantly reduces the number of trainable parameters, providing a built-in regularization effect. Decomposed convolutional layer showing matrices A and B forming intermediate matrix M
  2. Cross-Modal Pre-training: These factor matrices are first trained on large-scale RGB datasets to capture "common" object features (like shapes and boundaries) that are shared across modalities.
  3. Capacity Augmentation: To adapt to IR, the authors expand the rank of these matrices by a small amount ($\Delta r$). This creates a small, parallel trainable branch specifically for the IR modality. Capacity augmentation adding a parallel branch for the new modality
  4. Frozen Transfer: During IR training, the original RGB-trained weights are frozen. Only the tiny augmented branch is trained. This prevents the model from "forgetting" general object knowledge while allowing it to learn specific thermal cues.
  5. Complementarity Loss: A specialized loss function ($L_c$) is used to maximize the distance between the features learned by the RGB branch and the IR branch, ensuring they capture unique, non-redundant information.
Figure 2: Decomposed convolutional layer.
Capacity augmentation adding a parallel branch for the new modality

Datasets

  • FLIR Aligned RGB: Used for pre-training the base model to capture general object detection cues.
  • FLIR ADAS v1 IR: The target dataset for infrared detection. To simulate a "data-scarce" environment, the researchers conducted experiments using only 1% of the training set (just 62 images).

Key Insights and Ideas

  • Compression as Regularization: By using low-rank factorization, the model is forced to learn more compact and generalizable features, which is crucial when training data is limited.
  • Residual Modality Learning: The IR-specific branch acts similarly to a "skip-connection" or residual mapping. It doesn't need to learn how to detect a "car" from scratch; it only needs to learn the thermal "delta"—what makes a car look different in IR versus RGB.
  • Efficiency: The method achieves better results with significantly fewer trainable parameters in the target modality (e.g., a 95% reduction in trainable parameters for the IR branch compared to the baseline).

Key Results

  • Quantitative Gains: On the data-scarce (1% training) IR task, TensorFact achieved a 0.6205 mAP 50, outperforming the state-of-the-art YOLOv7 baseline (0.5849) by roughly 4%.
  • RGB Improvement: Surprisingly, the TensorFact decomposition also improved performance on the original RGB task, suggesting that the low-rank constraint helps even in data-rich scenarios.
  • Qualitative Superiority: Visualizations show that TensorFact is much better at detecting small, distant objects and objects that have low contrast against their background compared to standard baselines. Qualitative comparison showing TensorFact capturing more objects with higher precision than the baseline
Figure 1: Qualitative comparison of object detections by a state-of-the-art object detector (denoted as baseline) [51] and our TensorFact method on IR images. The orange, cyan, green boxes denote bicycle, person, and car classes respec- tively, while the associated numbers denote the confidence score of the prediction. The visualizations show that our proposed approach is better at capturing more objects, espe- cially those that are of a smaller size, with higher precision.

Limitations and Future Work

The authors acknowledge that while TensorFact is highly effective, the current work focuses primarily on the YOLOv7 architecture. Future research could explore:

  • Applying TensorFact to other backbone architectures and more complex computer vision tasks beyond object detection.
  • Investigating more sophisticated tensor decomposition formats (like Tensor-Train or Tucker decomposition) to see if they offer even better compression-performance trade-offs.
  • Testing the limits of the "data-scarce" scenario with even fewer than 62 images.

Summary updated

Abstract

The primary bottleneck towards obtaining good recognition performance in IR images is the lack of sufficient labeled training data, owing to the cost of acquiring such data. Realizing that object detection methods for the RGB modality are quite robust (at least for some commonplace classes, like person, car, etc.), thanks to the giant training sets that exist, in this work we seek to leverage cues from the RGB modality to scale object detectors to the IR modality, while preserving model performance in the RGB modality. At the core of our method, is a novel tensor decomposition method called TensorFact which splits the convolution kernels of a layer of a Convolutional Neural Network (CNN) into low-rank factor matrices, with fewer parameters than the original CNN. We first pretrain these factor matrices on the RGB modality, for which plenty of training data are assumed to exist and then augment only a few trainable parameters for training on the IR modality to avoid over-fitting, while encouraging them to capture complementary cues from those trained only on the RGB modality. We validate our approach empirically by first assessing how well our TensorFact decomposed network performs at the task of detecting objects in RGB images vis-a-vis the original network and then look at how well it adapts to IR images of the FLIR ADAS v1 dataset. For the latter, we train models under scenarios that pose challenges stemming from data paucity. From the experiments, we observe that: (i) TensorFact shows performance gains on RGB images; (ii) further, this pre-trained model, when fine-tuned, outperforms a standard state-of-the-art object detector on the FLIR ADAS v1 dataset by about 4% in terms of mAP 50 score.
BibTeX
@inproceedings{sharma2023tensor,
  author = {Manish Sharma and Moitreya Chatterjee and Kuan-Chuan Peng and Suhas Lohit and Michael Jones},
  title = {{Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection}},
  booktitle = {Workshops @ ICCV},
  year = {2023},
  doi = {10.1109/iccvw60793.2023.00099},
}
Figure