Learning Spatial-Frequency Transformer for Visual Object Tracking

Chuanming Tang, Xiao Wang, Yuanchao Bai, Zhe Wu, Jianlin Zhang, Yongmei Huang

Abstract

Transformer-based models have revolutionized visual object tracking (VOT) by capturing long-range dependencies. However, they suffer from two inherent limitations: the loss of spatial priors due to sequence flattening and the "low-pass filter" effect of self-attention, which leads to feature over-smoothing. We introduce the Spatial-Frequency Transformer (SFTransT), a framework designed to re-inject spatial awareness and amplify high-frequency details. By utilizing a Gaussian Generation Network (GGN) for dynamic spatial biasing and a High-frequency emphasis Attention (GPHA) module, SFTransT achieves state-of-the-art performance on benchmarks like GOT-10K and LaSOT while operating at real-time speeds.

Introduction

The transition from Convolutional Neural Networks (CNNs) to Transformers in VOT has brought significant gains in modeling global context. Yet, this shift introduces subtle architectural drawbacks. First, flattening 2D patches into 1D sequences discards the spatial structure inherent in images, forcing models to redundantly re-learn local relationships. Second, deep Transformer layers often act as low-pass filters, suppressing high-frequency signals like sharp edges and fine textures that are critical for distinguishing targets from distractors. SFTransT addresses these issues by treating the Transformer as a signal processor that preserves spatial and frequency-domain integrity.

Methodology

The overall architecture of SFTransT, featuring the SwinTransformer backbone, Interactive Learning module, and the core Spatial-Frequency Transformer.

Architecture Overview

The SFTransT pipeline consists of four integrated stages:

  1. Cross-Scale Fusion Backbone: Utilizing a Swin-Tiny backbone, the model employs a Cross-Scale Fusion (CSF) module to adaptively merge multi-scale features, ensuring robust representation before tracking begins.
  2. Interactive Learning: A Multi-Head Cross-Attention (MHCA) module facilitates deep interaction between the template object and the search region.
  3. Spatial-Frequency Transformer (SFFormer): This core component introduces two innovations:
  4. Gaussian Generation Network (GGN): Dynamically predicts a Gaussian spatial map to bias the attention matrix, prioritizing the target's spatial center.
  5. High-frequency emphasis Attention (GPHA): Decomposes attention weights into low-pass (DC) and high-frequency components. A learnable parameter ($\beta$) amplifies the high-frequency signals, preventing the over-smoothing typical of deep Transformers.
  6. Tracking Head: An MLP-based head performs classification using elliptical ground-truth labels and bounding box regression.

Experimental Results

SFTransT was evaluated on nine diverse benchmarks, including LaSOT, GOT-10K, and TrackingNet. The model demonstrates a superior balance of accuracy and efficiency, achieving a 72.7% Average Overlap (AO) on GOT-10K and a 69.1% Success Rate (SR) on LaSOT.

Qualitative comparison of attention maps between TransT and SFTransT. Our model shows sharper, more localized attention on critical object features.

Key Insights

  • Signal Processing Perspective: By recognizing self-attention's tendency to dampen high-frequency signals, the authors successfully applied all-pass filter concepts to deep learning.
  • Ablation Studies: Removing the GPHA module resulted in a 4.2% drop in AO on GOT-10K, highlighting the necessity of frequency management in deep tracking architectures.
  • Efficiency: Despite its complexity, the model contains only 29.6M parameters and maintains a real-time inference speed of 27 FPS on an NVIDIA RTX 3090.

Conclusion

SFTransT effectively mitigates the structural and spectral weaknesses of standard Transformers in visual tracking. By integrating dynamic spatial priors and high-frequency amplification, the tracker provides a robust solution for complex scenarios involving distractors and fine-grained textures. Future work will explore extending these spatial-frequency insights into the temporal domain to better model object motion over time.

Summary updated

BibTeX
@article{tang2023learning,
  author = {Chuanming Tang and Xiao Wang and Yuanchao Bai and Zhe Wu and Jianlin Zhang and Yongmei Huang},
  title = {{Learning Spatial-Frequency Transformer for Visual Object Tracking}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2023},
  url = {https://openreview.net/forum?id=pm5ZADDkgk},
}
Figure