Prompting for Multi-Modal Tracking

Jinyu Yang, Zhe Li, Feng Zheng, Ales Leonardis, Jingkuan Song

Introduction

Object tracking is significantly more robust when it can leverage multiple modalities, such as thermal cameras for low-visibility conditions or depth sensors to distinguish targets from cluttered backgrounds. However, while RGB tracking has flourished due to massive datasets like LaSOT, multi-modal tracking remains constrained by a "data-hungry bottleneck." Traditional approaches require complex fusion modules that are often poorly trained on small, specialized datasets, leading to a loss of the discriminative power learned from large-scale RGB data.

ProTrack introduces a "visual prompting" paradigm that flips the script: instead of adapting the model to multi-modal data, it adapts the data to a frozen, state-of-the-art RGB tracker. By "dyeing" auxiliary modalities and blending them into the input, ProTrack achieves state-of-the-art performance without any extra training.

The Challenge of Multi-Modal Tracking

Existing multi-modal trackers typically rely on early, middle, or late fusion strategies. These approaches face two major hurdles:

  1. Data Deficiency: Multi-modal datasets are often orders of magnitude smaller than RGB ones, leading to under-optimized fusion modules.
  2. Knowledge Forgetting: Fine-tuning a powerful pre-trained RGB tracker on a tiny multi-modal dataset often "breaks" the robust features the model learned from millions of RGB frames.

The ProTrack Framework

The core of ProTrack is a multi-modal prompt function. It takes the visible modality ($V$) and an auxiliary modality ($A$, like depth or thermal) and combines them into a single 3-channel image that appears as standard RGB data to a frozen tracker.

The ProTrack framework: multi-modal inputs are prompted into a single modality for a frozen RGB tracker.

The prompt function is defined as: $$f(V, A) = \lambda \cdot Color(A) + (1 - \lambda) \cdot Color(V)$$

Key components include:

  • Dyeing ($Color$): Auxiliary data (often single-channel) is transformed into 3-channel "colored" images using colormaps like JET. This makes the auxiliary data more salient and "familiar" to an RGB-trained model.
  • Weighting ($\lambda$): A very small weight (typically $\lambda = 0.05$) is used for the auxiliary prompt. This ensures the rich information of the RGB frame is preserved while "marking" it with useful cues from the other modality.
  • Frozen Backbone: ProTrack uses a state-of-the-art RGB tracker (specifically STARK) and keeps it entirely frozen. No multi-modal training is required.

Distribution Alignment

A primary insight of this work is that prompting acts as a bridge between domains. By "dyeing" thermal or depth maps, the authors effectively pull these disparate data distributions closer to the color domain the tracker was trained on. This allows the model to leverage its existing discriminative ability while gaining assistance from auxiliary data.

Visualizing how prompts reduce the distribution gap between auxiliary domains and the color domain.

Experimental Evaluation

Despite having zero training on multi-modal data, ProTrack achieved remarkable results across five diverse benchmarks:

  • RGB-Depth (RGB-D): On CDTB, it reached a new state-of-the-art F-score of 75.7%. On DepthTrack, it achieved 57.8%, surpassing specialized trackers trained specifically on that data.
  • RGB-Thermal (RGB-T): On LasHeR, it outperformed the well-known DMCNet by 4.9% in precision and 6.3% in success rate.
  • Versatility: The framework proved effective across various pre-trained architectures (e.g., DiMP, ATOM, KeepTrack), consistently improving their multi-modal performance.

Discussion and Future Work

While ProTrack is highly effective for dense modalities like depth and thermal, its performance on sparse data (like RGB-Event) is less dominant due to the lower information density of "dyed" binary event maps. Additionally, in extremely complex scenes, the prompt can occasionally disturb the input enough to confuse the tracker. Future research may explore more robust, learnable prompt designs and extend the framework to triple-modal tracking (RGB + Depth + Thermal).

Summary updated

BibTeX
@inproceedings{yang2022prompting,
  author = {Jinyu Yang and Zhe Li and Feng Zheng and Ales Leonardis and Jingkuan Song},
  title = {{Prompting for Multi-Modal Tracking}},
  booktitle = {ACM Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=HhKiAnArEv2},
}

Loading related content...

Figure