Abstract
Visual Object Tracking (VOT) has traditionally relied on Siamese networks that perform pair-wise matching between a target template and a search frame. While effective, this approach often neglects rich temporal context or relies on complex, hand-crafted update modules. VideoTrack addresses these limitations by shifting tracking to sequence-level modeling using a dedicated Video Transformer. By introducing a "disentangled dual-template" mechanism and a hierarchical Triplet-Block architecture, VideoTrack achieves state-of-the-art performance on benchmarks like GOT-10k (72.9% AO) and LaSOT (70.2% AUC) while operating at real-time speeds (~81 FPS).
Introduction
The dominant paradigm in VOT involves matching a static template from the first frame against a search region in subsequent frames. To account for appearance changes over time—such as rotation, illumination shifts, or occlusions—researchers have historically added "bells and whistles" like online template updating or memory queues. However, these additions often introduce significant computational overhead and numerous hyperparameters.
VideoTrack proposes a simpler alternative: an end-to-end video model that treats tracking as a spatiotemporal feature learning problem. Instead of augmenting a 2D image model, it "lifts" the matching pipeline into the 3D domain, as shown in Figure 1.
Methodology
The core of VideoTrack is a modified Vision Transformer (ViT) architecture designed for efficient spatiotemporal modeling.
The Triplet-Block and Hierarchical Attention
The architecture is built around the Triplet-Block, which processes three distinct branches: the initial template ($z$), intermediate "memory" frames ($t$), and the current search frame ($x$). As illustrated in Figure 2, these blocks use hierarchical video attention to mix information across the sequence.
Disentangled Dual-Template Mechanism
A key innovation is the decoupling of static and dynamic appearance. The first frame ($z$) provides a reliable anchor of the target's original appearance, preventing drift. Meanwhile, intermediate frames ($t$) capture the object's evolving state. The model uses cross-attention to aggregate dynamic clues from these intermediate frames before passing a refined representation to the search frame.
Asymmetric Attention and Tokenization
To maintain the purity of the target representation, VideoTrack employs asymmetric attention. Spatiotemporal guidance flows from historical frames to the search frame, but the search frame does not influence the historical templates. Furthermore, the authors found that "separated frame embedding" (treating frames individually at the start) is superior to 3D "tubelet" convolutions, as it prevents premature feature mixing that can degrade spatial understanding.
Experiments and Results
VideoTrack was trained on standard large-scale datasets (COCO, LaSOT, GOT-10k, TrackingNet) and evaluated against state-of-the-art trackers.
Performance Benchmarks
- GOT-10k: VideoTrack achieved a 72.9% AO, a 1.9% improvement over the previous leader, OStrack.
- LaSOT: The model reached a top-rank AUC of 70.2%, outperforming strong Siamese competitors like TransT.
- Efficiency: Despite its sequence-level modeling, the disentangled attention pattern allows the model to run at ~81 FPS on a V100 GPU, making it highly practical for real-time use.
Robustness and Pre-training
Qualitative analysis indicates that VideoTrack is particularly robust to full occlusions. By "looking back" at multiple historical frames, it can re-acquire targets more effectively than pair-wise methods. Additionally, the authors noted that Masked Autoencoder (MAE) pre-training is significantly more effective than supervised ImageNet pre-training for this task, likely due to the more robust structural representations learned by MAE.
Conclusion
VideoTrack demonstrates that sequence-level modeling via a Video Transformer can simplify the tracking pipeline while improving accuracy and robustness. By replacing complex update modules with a clean, feedforward architecture and a disentangled dual-template mechanism, it sets a new standard for real-time visual object tracking. Future work may involve scaling to larger backbones or exploring even more efficient attention patterns to handle longer temporal sequences.
Summary updated