Abstract
RGBT tracking has attracted increasing attention since RGB and thermal infrared data have strong complementary advantages, which could make trackers all-day and all-weather work. However, how to effectively represent RGBT data for visual tracking remains unstudied well. Existing works usually focus on extracting modality-shared or modality-specific information, but the potentials of these two cues are not well explored and exploited in RGBT tracking. In this paper, we propose a novel multi-adapter network to jointly perform modality-shared, modality-specific and instance-aware target representation learning for RGBT tracking. To this end, we design three kinds of adapters within an end-to-end deep learning framework. In specific, we use the modified VGG-M as the generality adapter to extract the modality-shared target representations. To extract the modality-specific features while reducing the computational complexity, we design a modality adapter, which adds a small block to the generality adapter in each layer and each modality in a parallel manner. Such a design could learn multilevel modality-specific representations with a modest number of parameters as the vast majority of parameters are shared with the generality adapter. We also design instance adapter to capture the appearance properties and temporal variations of a certain target. Moreover, to enhance the shared and specific features, we employ the loss of multiple kernel maximum mean discrepancy to measure the distribution divergence of different modal features and integrate it into each layer for more robust representation learning. Extensive experiments on two RGBT tracking benchmark datasets demonstrate the outstanding performance of the proposed tracker against the state-of-the-art methods.
Introduction
Object tracking using both RGB (visible) and Thermal (infrared) data is powerful because the modalities are complementary: RGB provides rich texture and color, while Thermal works in total darkness and handles shadows well. However, existing RGBT trackers often struggle with two extremes. Two-stream networks typically lead to massive parameter redundancy and ignore shared structures like object boundaries. Conversely, single-stream networks fail to capture the unique "heterogeneous" properties of each sensor.
The proposed Multi-Adapter Network (MANet++) aims to bridge this gap by creating a network that extracts shared features, modality-specific features, and instance-specific features simultaneously without exploding the parameter count.
The Multi-Adapter Network (MANet++)
The core of MANet++ consists of three specialized "adapters" integrated into an end-to-end framework:
- Generality Adapter (GA): A shared backbone (modified VGG-M) that extracts features common to both modalities, such as shapes and layouts.
- Modality Adapter (MA): Small, parallel blocks added to each layer of the GA. These use very few parameters (only ~20% of the GA) to capture the unique nuances of RGB vs. Thermal data.
- Instance Adapter (IA): A module that adapts to the specific object being tracked using a dynamic fusion scheme to weight the importance of each modality based on the current scene.
To improve efficiency, the network integrates an adaptive RoIAlign layer, which allows the network to extract features for all candidate target samples from a single feature map. This architectural tweak resulted in an 8x speedup over previous conference iterations (MANet) on the GTOT dataset.
Hierarchical Divergence (HD) Loss
To ensure the adapters learn distinct and useful information, the authors introduced a Hierarchical Divergence (HD) loss. Based on Multiple Kernel Maximum Mean Discrepancy (MK-MMD), this loss "pushes" the network to minimize the difference between modalities in the GA (forcing it to learn shared traits) while maximizing the difference in the MA (forcing it to learn unique traits).
This loss-driven feature separation ensures that the sensors aren't just duplicating each other's work, but are instead providing complementary discriminative power.
Experimental Results
MANet++ was evaluated on three major RGBT tracking benchmarks: GTOT, RGBT234, and VOT-RGBT2019.
- State-of-the-Art Performance: MANet++ achieved a Precision Rate (PR) of 80.0% and a Success Rate (SR) of 55.4% on the RGBT234 dataset, outperforming all other compared RGBT and RGB-only trackers.
- Robustness: The tracker showed significant improvements in challenging scenarios like "Low Resolution," "Partial Occlusion," and "Thermal Crossover" (where the target's temperature matches the background).
- Speed: On the GTOT dataset, MANet++ reached 27.3 FPS, making it much closer to real-time performance than many competing deep-learning-based trackers.
Conclusion
MANet++ demonstrates that "plugging in" tiny modality-specific adapters into a shared backbone is an efficient way to handle heterogeneous data. While the use of RoIAlign significantly boosts speed, the authors note it may lead to a "weak representation ability" for very low-resolution objects because feature maps are pooled into a fixed size. Future work suggests exploring deeper backbones like ResNet and incorporating additional modalities such as depth data.
Summary updated