Representation Learning for Visual Object Tracking by Masked Appearance Transfer

Haojie Zhao, Dong Wang, Huchuan LuCVPR 2023 · DOI

Abstract

Visual object tracking requires distinguishing a specific instance from its background and other similar objects across video frames. While most trackers rely on ImageNet-pretrained features, these representations are optimized for category-level invariance rather than instance-level discrimination. We propose Masked Appearance Transfer (MAT), a self-supervised pre-training method that treats tracking as an appearance reconstruction task. By jointly encoding masked patches from a template and a search region, MAT learns to transfer the target's appearance across frames. Our results demonstrate that MAT-learned features are highly discriminative, allowing a simple tracker head to achieve state-of-the-art performance at high frame rates.

Introduction

The core challenge in visual object tracking is maintaining a robust representation of a target object despite changes in pose, lighting, and occlusion. Traditional deep learning approaches often utilize backbones pre-trained on ImageNet classification. However, classification objectives encourage the model to ignore fine-grained instance details in favor of broad category labels. In contrast, tracking necessitates the opposite: a "dog" must be distinguished from all other "dogs" in the scene.

Furthermore, recent tracking research has focused heavily on architectural innovations in the "head" or "neck" of the network. This paper shifts the focus back to representation learning, proposing a method to pre-train backbones specifically for the temporal matching task inherent in tracking.

Methodology: Masked Appearance Transfer (MAT)

The proposed MAT framework utilizes a Vision Transformer (ViT) backbone and an encoder-decoder structure. The training process is built on two primary innovations: joint masked encoding and a dual-task reconstruction objective.

Joint Masked Encoding

Unlike Siamese networks that process the template and search region in parallel branches, MAT feeds both images into a shared encoder simultaneously. Both the template and search region are divided into patches, and a significant portion (e.g., 25%) are randomly masked. The remaining visible patches from both images are processed together, allowing the self-attention mechanism to learn cross-image correspondences early in the feature extraction process.

The MAT framework (b) compared to a standard masked autoencoder (a).

The Appearance Transfer Task

The decoder is tasked with two goals. First, it must perform standard image reconstruction for the search region. Second, it must perform the Appearance Transfer Task: reconstructing the target's appearance in the search region using the encoded features derived from the template. This objective forces the encoder to capture discriminative appearance details that are invariant to the spatial shifts and transformations between the template and the search region.

Experimental Results

To evaluate the quality of the learned representations, the authors use a minimal tracker head consisting of a depth-wise correlation operator and a lightweight bounding box regressor. The model was evaluated on massive tracking benchmarks including LaSOT, TrackingNet, and GOT10k.

Benchmark Performance

Despite the simplicity of the tracker head, MAT-pretrained features achieved exceptional results:

  • LaSOT: Achieved a 67.8% AUC, outperforming complex trackers like TransT and STARK.
  • Efficiency: The model operates at approximately 120 FPS on an RTX3090, making it ideal for real-time tracking.
  • Frozen Backbone: When the backbone was frozen during tracker training, it still achieved 65.2% AUC on LaSOT, indicating that the self-supervised pre-training captures nearly all necessary information for the task.
Visualizations of the reconstruction task across various object categories.

Key Insights

The authors' ablation studies revealed several critical findings:

  1. ImageNet Accuracy vs. Tracking: High classification accuracy on ImageNet does not directly translate to superior tracking performance, highlighting the need for tracking-specific pre-training.
  2. Position Jitter: Introducing random spatial shifts during pre-training is essential to prevent the model from relying on spatial shortcuts, forcing it to learn robust appearance features instead.
  3. Data Scaling: The quality of the representation improves significantly as the volume of tracking data used for pre-training increases.

Conclusion and Future Work

Masked Appearance Transfer (MAT) provides a powerful self-supervised paradigm for learning tracking-centric representations. By reframing tracking as a cross-frame appearance reconstruction task, MAT decouples feature quality from tracker head complexity. Future work may explore integrating these features into online-updating frameworks to better handle extreme appearance changes over long sequences, as well as optimizing the masking ratios for even finer-grained matching.

Summary updated

BibTeX
@inproceedings{zhao2023representation,
  author = {Haojie Zhao and Dong Wang and Huchuan Lu},
  title = {{Representation Learning for Visual Object Tracking by Masked Appearance Transfer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01793},
}
Figure