Target-Aware Tracking with Long-Term Context Attention

Kaijie He, Canlong Zhang, Sheng Xie, Zhixin Li, Zhiwen WangAAAI 2023 · DOI

Introduction

Most modern deep trackers follow the "Siamese" paradigm, comparing a small target template against a larger search area. However, this approach often suffers from context blindness, suboptimal backbones pre-trained for classification rather than tracking, and static templates that fail to adapt to appearance changes. TATrack addresses these issues by introducing a "target-aware" backbone that integrates long-term context directly into the feature extraction process.

Architecture and Methodology

TATrack centers on two main innovations: the Long-Term Context Attention (LCA) module and a dynamic online update strategy.

Long-Term Context Attention (LCA)

The LCA module simultaneously processes three inputs: the initial target template, a historical "context" frame, and the current search frame. By alternating LCA layers with Shifted Window Attention (SWA) layers inside the backbone, the model performs matching (correlation) while it is still extracting features. This allows the backbone to become "target-aware," focusing on relevant features early in the pipeline.

The TATrack architecture integrates LCA and SWA layers within the backbone to enable target-aware feature extraction across multiple stages.

To help the transformer distinguish between these concatenated inputs, the authors implemented a specialized relative position encoding that treats the relationship between the template, context, and search frames independently.

Dynamic Online Update (p-mean)

To handle appearance changes without massive computational overhead, the authors developed a "p-mean" (penalty-mean) algorithm. It calculates a threshold based on the historical average of classification confidence scores. By penalizing lower confidence scores, the tracker avoids updating its memory with poor-quality templates (e.g., during occlusion), preventing model drift.

Experimental Results

The model was evaluated on major benchmarks including LaSOT, TrackingNet, and GOT-10k. TATrack sets a new performance bar, particularly on long-term tracking tasks.

  • LaSOT: Achieved 71.1% AUC, outperforming previous leaders like MixFormer (67.9%) and SwinTrack (69.6%).
  • GOT-10k: Reached 79.2% AO (Average Overlap), a significant jump over existing state-of-the-art models.
Performance comparison on TrackingNet and LaSOT benchmarks, showing TATrack variants (B and L) achieving superior accuracy compared to existing state-of-the-art trackers.

The authors provide three versions of the model to balance accuracy and speed. TATrack-B (Base) offers high accuracy at approximately 14 FPS, while TATrack-S (Small) provides a faster alternative at ~30 FPS for real-time applications.

Conclusion and Future Work

TATrack demonstrates that matching during feature extraction leads to more discriminative features and better distractor suppression. While the "p-mean" logic improves robustness, the computational overhead of the large transformer model remains a challenge for edge devices, with the Large variant running at 6.6 FPS. Future work will focus on optimizing transformer modules and exploring lower-cost online update methods to maintain high accuracy while boosting frame rates.

Summary updated

BibTeX
@inproceedings{he2023target,
  author = {Kaijie He and Canlong Zhang and Sheng Xie and Zhixin Li and Zhiwen Wang},
  title = {{Target-Aware Tracking with Long-Term Context Attention}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25155},
}
Figure