TL;DR
The researchers present MANet++, a multi-adapter framework for RGBT (Visible + Thermal) tracking that effectively balances shared and modality-specific information. By using a novel Hierarchical Divergence (HD) loss and a parallel adapter architecture, the tracker achieves state-of-the-art accuracy while running significantly faster than previous iterations.
What Problem Does This Paper Address?
Object tracking using both RGB (visible) and Thermal (infrared) data is powerful because the modalities are complementary: RGB provides rich texture and color, while Thermal works in total darkness and handles shadows well.
However, existing RGBT trackers struggle with two extremes:
- Two-stream networks: These use separate branches for each modality, often leading to massive parameter redundancy and ignoring the fact that RGB and Thermal images share common structures (like object boundaries).
- Single-stream networks: These use the same parameters for both, which models shared information well but fails to capture the unique "heterogeneous" properties of each sensor.
The authors aim to bridge this gap by creating a network that extracts shared features, modality-specific features, and instance-specific features simultaneously without exploding the parameter count.
Methods
The core of the paper is the Multi-Adapter Network (MANet++), which consists of three specialized "adapters":
- Generality Adapter (GA): A shared backbone (modified VGG-M) that extracts features common to both modalities, such as shapes and layouts.
- Modality Adapter (MA): Small, parallel blocks added to each layer of the GA. These use very few parameters (only ~20% of the GA) to capture the unique nuances of RGB vs. Thermal data.
- Instance Adapter (IA): A module that adapts to the specific object being tracked. It uses a dynamic fusion scheme to weight the importance of each modality based on the current scene (e.g., giving more weight to Thermal if the RGB frame is blurry or dark).
To make these adapters work, the authors introduced a Hierarchical Divergence (HD) loss. Based on Multiple Kernel Maximum Mean Discrepancy (MK-MMD), this loss "pushes" the network to minimize the difference between modalities in the GA (forcing it to learn shared traits) while maximizing the difference in the MA (forcing it to learn unique traits).
Finally, they integrated an adaptive RoIAlign layer, which allows the network to extract features for all candidate target samples from a single feature map, drastically improving computational efficiency.
Datasets
The authors evaluated MANet++ on three major RGBT tracking benchmarks:
- GTOT: 50 video sequences of spatially and temporally aligned RGB and Thermal pairs.
- RGBT234: A large-scale dataset with 234 sequences and ~200,000 frames, featuring challenges like motion blur and occlusion.
- VOT-RGBT2019: A dataset from the Visual Object Tracking challenge consisting of 60 sequences.
Key Insights and Ideas
- The Power of Parallelism: Instead of building two giant networks, the authors show that "plugging in" tiny modality-specific adapters into a shared backbone is sufficient. This keeps the model lightweight while maintaining high discriminative power.
- Loss-Driven Feature Separation: The HD loss is the "secret sauce." Without it, the network might not naturally separate shared vs. specific information. By explicitly rewarding divergence in the Modality Adapters, they ensure the sensors aren't just duplicating each other's work.
- Efficiency Matters: By switching to RoIAlign, the authors achieved an 8x speedup over their previous conference version (MANet) on the GTOT dataset, proving that architectural tweaks for speed don't have to come at the cost of accuracy.
Key Results
- State-of-the-Art Performance: MANet++ achieved a Precision Rate (PR) of 80.0% and a Success Rate (SR) of 55.4% on the RGBT234 dataset, outperforming all other compared RGBT and RGB-only trackers.
- Robustness: The tracker showed significant improvements in challenging scenarios like "Low Resolution," "Partial Occlusion," and "Thermal Crossover" (where the target's temperature matches the background).
- Speed: On the GTOT dataset, MANet++ reached 27.3 FPS, making it much closer to real-time performance than many competing deep-learning-based trackers.
Limitations and Future Work
The authors note that while RoIAlign significantly boosts speed, it may lead to a "weak representation ability" for very low-resolution objects because the feature maps are pooled into a fixed size (e.g., 3x3). This was evident in the VOT-RGBT2019 results, where a version without RoIAlign actually performed slightly better on accuracy but was much slower.
For future work, the authors suggest exploring even deeper backbones (like ResNet) and incorporating additional modalities, such as depth or near-infrared data, into the multi-adapter framework.
Summary updated