Learning Spatial-Frequency Transformer for Visual Object Tracking

Chuanming Tang, Xiao Wang, Yuanchao Bai, Zhe Wu, Jianlin Zhang, Yongmei Huang

TL;DR

SFTransT is a novel visual tracking framework that addresses two major flaws in standard Transformer-based trackers: the loss of spatial information during feature flattening and the "low-pass filter" nature of self-attention that blurs fine details. By integrating Gaussian spatial priors and a high-frequency emphasis mechanism, SFTransT achieves state-of-the-art performance across nine major benchmarks while maintaining real-time speeds.

What Problem Does This Paper Address?

Visual Object Tracking (VOT) has recently shifted from convolutional neural networks (CNNs) to Transformers. While Transformers excel at capturing long-range dependencies, they introduce two subtle but significant problems:

  1. Spatial Information Loss: Standard Transformers flatten 2D image features into 1D sequences. This process destroys the inherent spatial structure of the target, ignoring the "spatial prior"—the common-sense knowledge that the target is likely concentrated in a specific region.
  2. The Low-Pass Filter Effect: Recent research suggests that the self-attention mechanism acts as a low-pass filter. In deep, stacked Transformer layers, this leads to "over-smoothing," where high-frequency information (like sharp edges and fine textures) is suppressed, leaving only the low-frequency "DC" components. For tracking, these high-frequency details are often what distinguish a target from a similar-looking distractor.

Methods

The authors propose SFTransT, a Siamese-style tracker built on a fully Transformer-based architecture. Its innovation lies in three main areas:

  • Cross-Scale Fusion (CSF) Backbone: Instead of using just the final layer of a backbone, they use a Swin-Tiny Transformer and adaptively fuse features from multiple stages. This ensures the model retains both high-level semantics and low-level structural details.
  • Gaussian Prior Attention (GPA): To re-introduce spatial awareness, the model uses a Gaussian Generation Network (GGN). This sub-network predicts a 2D Gaussian map that acts as a spatial bias, which is then injected directly into the Transformer's similarity matrix. This "reminds" the model to pay more attention to the central regions of interest.
  • High-Frequency Emphasis Attention (HEA): To combat over-smoothing, the authors decompose the attention weights into two branches: a DC (Direct Current) branch representing the average signal and an HF (High-Frequency) branch representing the details. They introduce a learnable parameter to amplify the HF component, effectively creating an "all-pass" filter that protects fine details through deep layers.
An overview of the SFTransT architecture, highlighting the feature extraction, interactive learning, and the core Spatial-Frequency Transformer module.

The core of these innovations is the SFFormer (Spatial-Frequency Transformer) block, which combines GPA and HEA into a single, powerful attention mechanism.

Detailed architecture of the proposed Spatial-Frequency Transformer layer, showing how Gaussian priors and high-frequency branches are integrated.

Datasets

The model was trained on a combination of large-scale datasets: COCO, TrackingNet, LaSOT, and GOT-10k.

Evaluation was conducted on nine diverse benchmarks to ensure robustness:

  • Short-term: OTB100, NFS, GOT-10k, UAV123, TrackingNet, WebUAV-3M.
  • Long-term: LaSOT, LaSOT_ext.
  • Language-guided: TNL2K.

Key Insights and Ideas

  • Attention is Blurry: The realization that self-attention is inherently a low-pass filter is a crucial "aha" moment. By explicitly treating attention as a signal processing problem, the authors found a way to "sharpen" the model's focus.
  • Spatial Priors Matter: Even in the era of global attention, local spatial context is vital. Injecting a simple Gaussian bias helps the model converge faster and ignore irrelevant background clutter.
  • Edge-Awareness: Qualitative results show that SFTransT pays significantly more attention to target boundaries and specific components (like the head or tail of an animal) compared to previous SOTA trackers like TransT, which often focus on less discriminative central "trunks."
Comparison of activation maps showing how SFTransT focuses on discriminative edges and target parts compared to TransT.

Key Results

SFTransT sets new performance bars across multiple datasets:

  • GOT-10k: Achieved a state-of-the-art 72.7% Average Overlap (AO), outperforming the previous best (MixFormer) by 1.5%.
  • LaSOT: Obtained a Success Rate (SR) of 69.1%, a significant lead over other Transformer-based trackers.
  • Efficiency: Despite the added complexity, the model contains only 29.6M parameters and runs at 27 FPS on a single RTX 3090, making it suitable for real-time applications.
Table showing performance comparison on the GOT-10k benchmark, where SFTransT achieves the top score in AO and SR metrics.

Attribute analysis on LaSOT (Figure 4) further demonstrates that SFTransT consistently outperforms competitors across challenging scenarios like background clutter, occlusion, and deformation.

Attribute analysis on the LaSOT dataset, showing SFTransT's superior performance across various challenging tracking conditions.

Limitations and Future Work

While highly effective, the authors note that SFTransT still struggles with dense and very small-scale targets, where distractors are easily confused with the target.

Future work aims to:

  1. Incorporate temporal representations to explicitly model how a target's appearance changes over time.
  2. Explore spatio-temporal preserving representations and semi-supervised methods to better discriminate between targets and latent distractor patterns.

Summary updated

Abstract

Recently, some researchers have begun to adopt the Transformer to combine or replace the widely used ResNet as their new backbone network. As the Transformer captures the long-range relations between pixels well using the self-attention scheme, which complements the issues caused by the limited receptive field of CNN. Although their trackers work well in regular scenarios, they simply flatten the 2D features into a sequence to better match the Transformer. We believe these operations ignore the spatial prior of the target object, which may lead to sub-optimal results only. In addition, many works demonstrate that self-attention is actually a low-pass filter, which is independent of input features or keys/queries. That is to say, it may suppress the high-frequency component of the input features and preserve or even amplify the low-frequency information. To handle these issues, in this paper, we propose a unified Spatial-Frequency Transformer that models the Gaussian spatial Prior and High-frequency emphasis Attention (GPHA) simultaneously. To be specific, Gaussian spatial prior is generated using dual Multi-Layer Perceptrons (MLPs) and injected into the similarity matrix produced by multiplying Query and Key features in self-attention. The output will be fed into a softmax layer and then decomposed into two components, i.e., the direct and high-frequency signal. The low- and high-pass branches are rescaled and combined to achieve all-pass, therefore, the high-frequency features will be protected well in stacked self-attention layers. We further integrate the Spatial-Frequency Transformer into the Siamese tracking framework and propose a novel tracking algorithm termed SFTransT. The cross-scale fusion based SwinTransformer is adopted as the backbone, and also a multi-head cross-attention module is used to boost the interaction between search and template features. The output will be fed into the tracking head for target localization. Extensive experiments on short-term and long-term tracking benchmarks all demonstrate the effectiveness of our proposed framework. Source code will be released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Tchuanm/SFTransT.git</uri> .
BibTeX
@article{tang2023learning,
  author = {Chuanming Tang and Xiao Wang and Yuanchao Bai and Zhe Wu and Jianlin Zhang and Yongmei Huang},
  title = {{Learning Spatial-Frequency Transformer for Visual Object Tracking}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2023},
  url = {https://openreview.net/forum?id=pm5ZADDkgk},
}
Figure