ATSTrack: Enhancing Visual-Language Tracking by Aligning Temporal and Spatial Scales

Yihao Zhen, Qiang Wang, Yu Qiao, Liangqiong Qu, Huijie Fan

Abstract

A main challenge of Visual-Language Tracking (VLT) is the misalignment between visual inputs and language descriptions caused by target movement. Previous trackers have explored many effective feature modification methods to preserve more aligned features. However, an important yet unexplored factor ultimately hinders their capability, which is the inherent differences in the temporal and spatial scale of information between visual and language inputs. To address this issue, we propose a novel visual-language tracker that enhances the effect of feature modification by aligning temporal and spatial scale of different input components, named as ATSTrack. Specifically, we decompose each language description into phrases with different attributes based on their temporal and spatial correspondence with visual inputs, and modify their features in a fine-grained manner. Moreover, we introduce a Visual-Language token that comprises modified linguistic information from the previous frame to guide the model to extract visual features that are more relevant to language description, thereby reducing the impact caused by the differences in spatial scale. Experimental results show that our proposed ATSTrack achieves performance comparable to existing methods.

Introduction: The Challenge of Misalignment

Visual-Language Tracking (VLT) aims to follow an object in a video using both an initial bounding box and a natural language description (e.g., "a white dog running on grass"). While adding language should make trackers more robust, it introduces a significant challenge: misalignment.

Misalignment between description and visual input and the proposed solution.

This misalignment happens in two primary ways:

  1. Temporal Misalignment: A description might say "running," but later in the sequence, the target may be "sitting." The static text no longer matches the dynamic video.
  2. Spatial Misalignment: Language is "sparse"—a single word like "dog" refers to a small cluster of pixels. Visual data is "dense"—the frame contains millions of pixels, most of which are irrelevant background.

Previous trackers often treated visual and language features as monolithic blocks, leading to "noisy" fusion where the background interferes with the target description.

Methodology: ATSTrack

ATSTrack addresses these scale differences through a "divide and conquer" strategy, decomposing language into specific attributes and modulating features in a fine-grained manner.

The architecture of ATSTrack featuring Fine-grained modulation.

Attribute Decomposition and Fine-Grained Modulation (FGM)

Using a Large Language Model (LLM), the system splits a description into four distinct categories: Category (what it is), Appearance (how it looks), Action (what it’s doing), and Location (where it is). ATSTrack then applies specific modules:

  • Visual Feature Modification (VFM): Uses the "Category" feature to "clean" the template image, suppressing background pixels so the model focuses only on the object.
  • Language Feature Ablation (LFA): A gating mechanism that identifies when a "Location" description (like "near the tree") no longer matches the visual search area and effectively "mutes" that part of the text to prevent confusion.

The Visual-Language (VL) Token

The model generates a special token that carries modified linguistic information from the previous frame and injects it directly into the visual backbone of the current frame. This acts as a "guide," telling the visual encoder exactly which features (colors, shapes, or actions) are relevant before the final tracking decision is made.

Experimental Evaluation

The model was trained and evaluated on several major benchmarks, including TNL2K, LaSOT, and $OTB_{lang}$.

Key Results

  • State-of-the-Art Performance: On TNL2K, ATSTrack achieved an AUC of 66.2%, surpassing the previous state-of-the-art by roughly 3%.
  • Robustness: In qualitative tests, ATSTrack successfully tracked targets through severe occlusions and fast motions where baseline trackers (like OSTrack) failed.
  • Attribute Performance: On the LaSOT dataset, the model showed superior performance across almost all challenge categories, including background clutter and viewpoint changes.

Insights and Future Directions

The core innovation of ATSTrack is recognizing that not all words are created equal. "Action" features need to be compared against a sequence of video frames (temporal), while "Appearance" features primarily relate to the latest frame (spatial). Treating them differently prevents the model from getting distracted.

While ATSTrack is highly effective, the authors note that the attribute decomposition currently relies on an offline LLM. Integrating a lightweight, real-time language processor could make the system more flexible for "in-the-wild" use. Future work will also explore refining how temporal relationships are modeled between the VL token and the video stream to further improve long-term tracking.

Summary updated

Abstract

A main challenge of Visual-Language Tracking (VLT) is the misalignment between visual inputs and language descriptions caused by target movement. Previous trackers have explored many effective feature modification methods to preserve more aligned features. However, an important yet unexplored factor ultimately hinders their capability, which is the inherent differences in the temporal and spatial scale of information between visual and language inputs. To address this issue, we propose a novel visual-language tracker that enhances the effect of feature modification by \textbf{A}ligning \textbf{T}emporal and \textbf{S}patial scale of different input components, named as \textbf{ATSTrack}. Specifically, we decompose each language description into phrases with different attributes based on their temporal and spatial correspondence with visual inputs, and modify their features in a fine-grained manner. Moreover, we introduce a Visual-Language token that comprises modified linguistic information from the previous frame to guide the model to extract visual features that are more relevant to language description, thereby reducing the impact caused by the differences in spatial scale. Experimental results show that our proposed ATSTrack achieves performance comparable to existing methods. Our code will be released.
BibTeX
@misc{zhen2025atstrack,
  author = {Yihao Zhen and Qiang Wang and Yu Qiao and Liangqiong Qu and Huijie Fan},
  title = {{ATSTrack: Enhancing Visual-Language Tracking by Aligning Temporal and Spatial Scales}},
  year = {2025},
  eprint = {2507.00454},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.00454},
}
Figure