Exploring Enhanced Contextual Information for Video-Level Object Tracking

Ben Kang, Xin Chen, Simiao Lai, Yang Liu, Yi Liu, Dong WangAAAI 2025 · DOI

Introduction

Visual object tracking is a fundamental task in computer vision, typically involving the matching of a target template against a search region in subsequent video frames. While image-level trackers have seen significant success, they often struggle with complex scenarios such as long-term occlusions, rapid appearance changes, or motion blur. These challenges arise because image-level methods treat each frame in isolation, failing to leverage the rich temporal context available in a video stream.

Recent attempts to create "video-level" trackers have introduced "extra tokens" to carry information across frames. However, as illustrated in the comparison below, these tokens often create an information bottleneck. A fixed number of tokens cannot capture the full complexity of a target's history, leading to significant context loss over time.

Comparison of temporal context transmission methods: (a) Dynamic Template, (b) Extra Token, and (c) the proposed Hidden State approach.

Methodology: The MCITrack Framework

To overcome the limitations of token-based memory, the authors propose MCITrack, a framework that utilizes the hidden states of a State Space Model (SSM)—specifically Mamba—to store and transmit contextual information. Unlike Transformers, which have quadratic complexity relative to sequence length, Mamba offers linear scaling, making it an ideal candidate for processing long-range temporal dependencies in video.

Contextual Information Fusion (CIF) Module

The core innovation of MCITrack is the Contextual Information Fusion (CIF) module. This module is deeply integrated into the Transformer backbone (Fast-iTPN), with a CIF block paired with each backbone block. This multi-level integration allows the model to fuse context at various feature scales, from low-level textures to high-level semantics.

Overview of the MCITrack framework and the internal structure of the Contextual Information Fusion (CIF) block.

As shown in the architecture diagram, each CIF block consists of three primary components:

  1. Mamba Layer: Acts as the temporal memory, maintaining a hidden state ($h$) that evolves as the video progresses.
  2. In-Attention Layer: A cross-attention mechanism that injects the stored historical context from the Mamba layer into the current frame's backbone features.
  3. Out-Attention Layer: A mechanism that extracts relevant features from the current frame to update the Mamba hidden state, ensuring the memory remains current.

Selective State Updates

A critical challenge in online tracking is preventing "model drift," where the tracker learns from incorrect information (e.g., when the target is occluded). MCITrack addresses this by implementing a selective update strategy: the hidden states are only updated if the classification confidence score for the current frame exceeds a predefined threshold. This ensures that only high-quality, reliable information is stored in the temporal memory.

Experimental Results and Analysis

MCITrack was evaluated across eight major benchmarks, including LaSOT, GOT-10k, and TrackingNet. The model was trained on a diverse set of datasets such as LaSOT, GOT-10k, TrackingNet, COCO, and VastTrack.

State-of-the-Art Performance

The results demonstrate that MCITrack sets new performance ceilings for video-level tracking:

  • LaSOT: The large variant (MCITrack-L384) achieved a 76.6% AUC, representing a 2.6% improvement over the previous state-of-the-art, ODTrack.
  • GOT-10k: The model reached a milestone 80.0% Average Overlap (AO), showcasing its robustness in general object tracking.
  • Efficiency and Speed: Despite the added complexity of temporal modeling, MCITrack remains highly efficient. The "Base" model (MCITrack-B224) runs at 35 FPS on a standard NVIDIA 2080 Ti GPU, while the "Tiny" version reaches 51 FPS, making it suitable for real-time deployment.

Key Insights

Ablation studies revealed that the length of the video clip used during training is crucial. The authors found that a 5-frame clip provides the optimal balance; shorter clips lack sufficient context, while longer clips increase the learning burden without providing proportional gains. Furthermore, the transition from explicit tokens to implicit hidden states proved to be the decisive factor in capturing richer, more flexible temporal representations.

Conclusion

MCITrack represents a significant shift in video-level object tracking. By replacing limited extra tokens with the high-capacity hidden states of Mamba, the framework effectively bridges the gap between image-level matching and long-term temporal reasoning. Its deep integration of contextual information and efficient linear scaling make it a powerful and practical solution for real-world tracking challenges.

Summary updated

BibTeX
@inproceedings{kang2025exploring,
  author = {Ben Kang and Xin Chen and Simiao Lai and Yang Liu and Yi Liu and Dong Wang},
  title = {{Exploring Enhanced Contextual Information for Video-Level Object Tracking}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i4.32440},
}
Figure