Introduction
Object tracking is significantly more robust when it can leverage multiple modalities, such as thermal cameras for low-visibility conditions or depth sensors to distinguish targets from cluttered backgrounds. However, while RGB tracking has flourished due to massive datasets like LaSOT, multi-modal tracking remains constrained by a "data-hungry bottleneck." Traditional approaches require complex fusion modules that are often poorly trained on small, specialized datasets, leading to a loss of the discriminative power learned from large-scale RGB data.
ProTrack introduces a "visual prompting" paradigm that flips the script: instead of adapting the model to multi-modal data, it adapts the data to a frozen, state-of-the-art RGB tracker. By "dyeing" auxiliary modalities and blending them into the input, ProTrack achieves state-of-the-art performance without any extra training.
The Challenge of Multi-Modal Tracking
Existing multi-modal trackers typically rely on early, middle, or late fusion strategies. These approaches face two major hurdles:
- Data Deficiency: Multi-modal datasets are often orders of magnitude smaller than RGB ones, leading to under-optimized fusion modules.
- Knowledge Forgetting: Fine-tuning a powerful pre-trained RGB tracker on a tiny multi-modal dataset often "breaks" the robust features the model learned from millions of RGB frames.
The ProTrack Framework
The core of ProTrack is a multi-modal prompt function. It takes the visible modality ($V$) and an auxiliary modality ($A$, like depth or thermal) and combines them into a single 3-channel image that appears as standard RGB data to a frozen tracker.
The prompt function is defined as: $$f(V, A) = \lambda \cdot Color(A) + (1 - \lambda) \cdot Color(V)$$
Key components include:
- Dyeing ($Color$): Auxiliary data (often single-channel) is transformed into 3-channel "colored" images using colormaps like JET. This makes the auxiliary data more salient and "familiar" to an RGB-trained model.
- Weighting ($\lambda$): A very small weight (typically $\lambda = 0.05$) is used for the auxiliary prompt. This ensures the rich information of the RGB frame is preserved while "marking" it with useful cues from the other modality.
- Frozen Backbone: ProTrack uses a state-of-the-art RGB tracker (specifically STARK) and keeps it entirely frozen. No multi-modal training is required.
Distribution Alignment
A primary insight of this work is that prompting acts as a bridge between domains. By "dyeing" thermal or depth maps, the authors effectively pull these disparate data distributions closer to the color domain the tracker was trained on. This allows the model to leverage its existing discriminative ability while gaining assistance from auxiliary data.
Experimental Evaluation
Despite having zero training on multi-modal data, ProTrack achieved remarkable results across five diverse benchmarks:
- RGB-Depth (RGB-D): On CDTB, it reached a new state-of-the-art F-score of 75.7%. On DepthTrack, it achieved 57.8%, surpassing specialized trackers trained specifically on that data.
- RGB-Thermal (RGB-T): On LasHeR, it outperformed the well-known DMCNet by 4.9% in precision and 6.3% in success rate.
- Versatility: The framework proved effective across various pre-trained architectures (e.g., DiMP, ATOM, KeepTrack), consistently improving their multi-modal performance.
Discussion and Future Work
While ProTrack is highly effective for dense modalities like depth and thermal, its performance on sparse data (like RGB-Event) is less dominant due to the lower information density of "dyed" binary event maps. Additionally, in extremely complex scenes, the prompt can occasionally disturb the input enough to confuse the tracker. Future research may explore more robust, learnable prompt designs and extend the framework to triple-modal tracking (RGB + Depth + Thermal).
Summary updated