Adaptive Perception for Unified Visual Multi-modal Object Tracking

Xiantao Hu, Bineng Zhong, Qihua Liang, Zhiyi Mo, Liangtao Shi, Ying Tai, Jian Yang

Introduction: Beyond the Dominance of RGB

In computer vision, the standard color (RGB) feed has traditionally been treated as the primary source of truth. Most multi-modal object trackers relegate thermal, depth, or event-based data to "auxiliary" status. However, this hierarchy fails in challenging environments—such as total darkness or smoke-filled rooms—where the RGB sensor becomes unreliable.

The paper "Adaptive Perception for Unified Visual Multi-modal Object Tracking" introduces APTrack, a unified tracker that challenges this paradigm. By treating all modalities as equals and employing an "adaptive perception" mechanism, APTrack tracks objects across diverse sensors without requiring task-specific fine-tuning.

The Problem: Rigidity and Fragmentation

Current multi-modal trackers often suffer from two major design flaws:

  1. Rigidity: A dominant-auxiliary design prevents auxiliary sensors from taking over when RGB quality is low (e.g., night scenes or motion blur).
  2. Fragmentation: Different sensor pairs (RGB+Thermal vs. RGB+Depth) typically require separate, fine-tuned models due to their distinct characteristics.

APTrack addresses these by using a unified parameter model—a single set of weights capable of handling any "RGB+X" task by autonomously determining which modality is most informative at any given time.

Methods: Equal Modeling and Adaptive Interaction

The core of APTrack is its Equal Modeling Strategy. Instead of an RGB-centric backbone, both modalities are processed through shared-weight Transformer blocks. This ensures that features from all sensors are treated with equal importance from the start.

To facilitate efficient communication between modalities, the authors developed the Adaptive Modality Interaction (AMI) module. As shown in Figure 1, the AMI operates in three distinct stages:

  • Token Learning: The model learns a small set of representative "tokens" for each sensor, drastically reducing the computational load compared to pixel-to-pixel comparisons.
  • Global Modal Perceptor: These tokens undergo cross-modal attention, allowing the modalities to align information and swap features.
  • Token Fusion: The enriched token information is projected back into the original feature maps.
The APTrack architecture featuring the Adaptive Modality Interaction (AMI) module for efficient cross-modal feature alignment.

Additionally, a Dual-Template Mechanism maintains both an initial target template and a dynamic one that updates based on high-confidence results, allowing the tracker to adapt to appearance changes over time.

Key Insights: Modality Agnosticism and Efficiency

  • Adaptive Signal Selection: By not hard-coding a dominant modality, the network learns to favor the sensor with the clearest signal. In low-light conditions, attention maps naturally shift toward Thermal or Depth features.
  • Sparsity for Speed: Cross-modal attention is typically expensive. By condensing features into learnable tokens, APTrack achieves global interaction while maintaining real-time speeds (50.5 FPS).
  • The Power of Unified Parameters: The success of APTrack suggests that the fundamental logic of tracking is consistent across modalities, provided the feature alignment is handled adaptively.

Results: State-of-the-Art Performance

APTrack consistently outperformed both unified and task-specific state-of-the-art (SOTA) models across all tested modalities.

Qualitative comparison of APTrack against other state-of-the-art trackers in challenging multi-modal scenarios, demonstrating its robustness in low-light and occluded environments.
  • Accuracy: On the LasHeR (RGB-T) dataset, it achieved a 58.9% Success rate, significantly surpassing the previous unified SOTA, Un-Track (51.3%).
  • Robustness: On VOT-RGBD2022, it reached an EAO (Expected Average Overlap) of 77.4, compared to 71.8 for the next best unified model.
  • Efficiency: Despite its complex interaction module, APTrack maintains a runtime of 50.5 FPS on an NVIDIA RTX 4090.
  • Ablation Study: Removing the Adaptive Perception structure led to a 4.3% drop in success, confirming that the "equal modeling" approach is the primary driver of its performance.

Limitations and Future Work

While APTrack represents a significant advancement, some challenges remain:

  • Computational Symmetry: Processing both modalities with high-level blocks increases total computational demand compared to models with lightweight auxiliary branches.
  • Expanding Modalities: The current version focuses on visual/spatial sensors. Future work aims to incorporate Natural Language prompts to create a truly universal tracker guided by both visual and linguistic cues.

Summary updated

BibTeX
@article{hu2025adaptive,
  author = {Xiantao Hu and Bineng Zhong and Qihua Liang and Zhiyi Mo and Liangtao Shi and Ying Tai and Jian Yang},
  title = {{Adaptive Perception for Unified Visual Multi-modal Object Tracking}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2025},
  url = {https://openreview.net/forum?id=vMvfphE2uD},
}
Figure