TL;DR
Researchers have developed TensorFact, a novel tensor decomposition method that enables high-performance object detection in infrared (IR) imagery even when labeled data is extremely scarce. By leveraging knowledge from abundant RGB datasets and using a parameter-efficient parallel branch to capture IR-specific cues, the method outperforms state-of-the-art baselines by 4% mAP while significantly reducing the risk of overfitting.
What Problem Does This Paper Address?
Object detection in standard RGB images has reached a high level of maturity, thanks to massive labeled datasets like COCO. However, for applications like autonomous driving, RGB cameras fall short in low-light, foggy, or nighttime conditions. Thermal Infrared (IR) cameras are ideal for these scenarios because they detect heat rather than visible light.
The major hurdle is data paucity: labeling IR images is expensive and time-consuming, leading to a lack of large-scale datasets. When we try to train modern, over-parameterized deep networks (like YOLOv7) on small IR datasets, they tend to overfit, losing their ability to generalize. Furthermore, standard domain adaptation techniques often fail because the "modality shift" between RGB (color/texture) and IR (heat signatures) is too extreme for simple transfer learning.
Methods
The authors introduce TensorFact, a method that reimagines how convolutional layers learn across different modalities.
- Tensor Decomposition: Instead of using standard 4D convolution kernels, TensorFact decomposes each kernel into two low-rank factor matrices ($A$ and $B$). This significantly reduces the number of trainable parameters, providing a built-in regularization effect. Decomposed convolutional layer showing matrices A and B forming intermediate matrix M
- Cross-Modal Pre-training: These factor matrices are first trained on large-scale RGB datasets to capture "common" object features (like shapes and boundaries) that are shared across modalities.
- Capacity Augmentation: To adapt to IR, the authors expand the rank of these matrices by a small amount ($\Delta r$). This creates a small, parallel trainable branch specifically for the IR modality. Capacity augmentation adding a parallel branch for the new modality
- Frozen Transfer: During IR training, the original RGB-trained weights are frozen. Only the tiny augmented branch is trained. This prevents the model from "forgetting" general object knowledge while allowing it to learn specific thermal cues.
- Complementarity Loss: A specialized loss function ($L_c$) is used to maximize the distance between the features learned by the RGB branch and the IR branch, ensuring they capture unique, non-redundant information.
Datasets
- FLIR Aligned RGB: Used for pre-training the base model to capture general object detection cues.
- FLIR ADAS v1 IR: The target dataset for infrared detection. To simulate a "data-scarce" environment, the researchers conducted experiments using only 1% of the training set (just 62 images).
Key Insights and Ideas
- Compression as Regularization: By using low-rank factorization, the model is forced to learn more compact and generalizable features, which is crucial when training data is limited.
- Residual Modality Learning: The IR-specific branch acts similarly to a "skip-connection" or residual mapping. It doesn't need to learn how to detect a "car" from scratch; it only needs to learn the thermal "delta"—what makes a car look different in IR versus RGB.
- Efficiency: The method achieves better results with significantly fewer trainable parameters in the target modality (e.g., a 95% reduction in trainable parameters for the IR branch compared to the baseline).
Key Results
- Quantitative Gains: On the data-scarce (1% training) IR task, TensorFact achieved a 0.6205 mAP 50, outperforming the state-of-the-art YOLOv7 baseline (0.5849) by roughly 4%.
- RGB Improvement: Surprisingly, the TensorFact decomposition also improved performance on the original RGB task, suggesting that the low-rank constraint helps even in data-rich scenarios.
- Qualitative Superiority: Visualizations show that TensorFact is much better at detecting small, distant objects and objects that have low contrast against their background compared to standard baselines. Qualitative comparison showing TensorFact capturing more objects with higher precision than the baseline
Limitations and Future Work
The authors acknowledge that while TensorFact is highly effective, the current work focuses primarily on the YOLOv7 architecture. Future research could explore:
- Applying TensorFact to other backbone architectures and more complex computer vision tasks beyond object detection.
- Investigating more sophisticated tensor decomposition formats (like Tensor-Train or Tucker decomposition) to see if they offer even better compression-performance trade-offs.
- Testing the limits of the "data-scarce" scenario with even fewer than 62 images.
Summary updated