Adaptive and Background-Aware Vision Transformer for Real-Time UAV Tracking

Shuiwang Li, Yangxiang Yang, Dan Zeng, Xucheng WangICCV 2023 · OpenReview

Abstract

While discriminative correlation filters (DCF)-based trackers prevail in UAV tracking for their favorable efficiency, lightweight convolutional neural network (CNN)-based trackers using filter pruning have also demonstrated remarkable efficiency and precision. However, the use of pure vision transformer models (ViTs) for UAV tracking remains unexplored. In this paper, we propose an efficient ViT-based tracking framework, Aba-ViTrack, for UAV tracking. In our framework, feature learning and template-search coupling are integrated into an efficient one-stream ViT to avoid an extra heavy relation modeling module. The proposed Aba-ViT exploits an adaptive and background-aware token computation method to reduce inference time. This approach adaptively discards tokens based on learned halting probabilities, which a priori are higher for background tokens than target ones. Extensive experiments on six UAV tracking benchmarks demonstrate that the proposed Aba-ViTrack achieves state-of-the-art performance in UAV tracking.

Introduction

Visual tracking for Unmanned Aerial Vehicles (UAVs) requires a delicate balance between high precision and computational efficiency. Drones operate in resource-constrained environments with limited battery life and onboard processing power, yet they must navigate complex scenarios involving extreme viewing angles, motion blur, and fast-moving targets.

Historically, the field has been divided between Discriminative Correlation Filters (DCF), which offer high speed but struggle with appearance changes, and Convolutional Neural Networks (CNNs), which provide high accuracy at the cost of significant computational overhead. While Vision Transformers (ViTs) have set new benchmarks in general computer vision, their application to real-time UAV tracking has been hindered by their "token-heavy" nature. Processing every image patch through every layer is inefficient, especially when large portions of the search area consist of irrelevant background. Aba-ViTrack addresses this by introducing a "smart" ViT that adaptively allocates computation.

Aba-ViTrack Framework

The Aba-ViTrack framework introduces two primary innovations to make ViTs viable for real-time aerial tracking: a one-stream backbone and background-aware token halting.

One-Stream ViT Backbone

Unlike traditional Siamese trackers that process the target template and search area in separate branches before fusion, Aba-ViTrack utilizes a one-stream architecture. The template and search patches are concatenated and fed into the transformer simultaneously. This allows the model to perform feature extraction and relation modeling (matching the target to the scene) concurrently, eliminating the need for heavy cross-correlation or "neck" modules.

Background-Aware Token Halting (Aba-ViT)

The core of the efficiency gain comes from the Aba-ViT mechanism, which predicts a halting probability for every token at each layer. If a token is identified as background noise, the model "halts" its processing early to save computational cycles.

Overview of the Aba-ViTrack framework, highlighting the one-stream backbone and the background-aware token halting mechanism.

To train this mechanism, the authors use a modified Ponder Loss that specifically penalizes the model for spending excessive time on background tokens. By leveraging prior knowledge of the target's location, the model learns to "glance" at the background while "pondering" the target, effectively acting as a regularizer that prevents the tracker from being distracted by noise.

Experimental Results

Aba-ViTrack was evaluated on six rigorous UAV-specific benchmarks, including UAV123, DTB70, and VisDrone2018, after training on large-scale datasets like GOT-10k and LaSOT.

Performance and Speed

On the UAV123 benchmark, Aba-ViTrack achieved a state-of-the-art precision of 0.864, outperforming previous CNN-based leaders like TCTrack. In terms of speed, the model reaches approximately 180 FPS on a desktop GPU (NVIDIA TitanX). Crucially for UAV applications, it maintains a real-time speed of 35.6 FPS on an NVIDIA Jetson TX2, an embedded edge device commonly used in drone payloads.

Comparison of precision vs. speed (fps) for Aba-ViTrack against state-of-the-art DCF and CNN-based trackers.

Ablation Studies

Ablation experiments confirmed that the "Background-Aware" weighting is vital, providing a 1.8% precision boost over standard adaptive ViT models. This demonstrates that targeted computational savings on background regions are more effective than uniform token reduction.

Conclusion

Aba-ViTrack represents the first pure Vision Transformer framework optimized for real-time UAV tracking. By integrating feature learning and relation modeling into a single stream and employing an intelligent token-halting mechanism, it achieves a superior balance of speed and accuracy. Future work may involve refining the distributional priors for token exiting and extending this background-aware logic to other computer vision tasks like object detection.

Summary updated

Abstract

While discriminative correlation filters (DCF)-based trackers prevail in UAV tracking for their favorable efficiency, lightweight convolutional neural network (CNN)-based trackers using filter pruning have also demonstrated remarkable efficiency and precision. However, the use of pure vision transformer models (ViTs) for UAV tracking remains unexplored, which is a surprising finding given that ViTs have been shown to produce better performance and greater efficiency than CNNs in image classification. In this paper, we propose an efficient ViT-based tracking framework, Aba-ViTrack, for UAV tracking. In our framework, feature learning and template-search coupling are integrated into an efficient one-stream ViT to avoid an extra heavy relation modeling module. The proposed Aba-ViT exploits an adaptive and background-aware token computation method to reduce inference time. This approach adaptively discards tokens based on learned halting probabilities, which a priori are higher for background tokens than target ones. Extensive experiments on six UAV tracking benchmarks demonstrate that the proposed Aba-ViTrack achieves state-of-the-art performance in UAV tracking. Code is available at https://github.com/xyyang317/Aba-ViTrack.
BibTeX
@inproceedings{li2023adaptive,
  author = {Shuiwang Li and Yangxiang Yang and Dan Zeng and Xucheng Wang},
  title = {{Adaptive and Background-Aware Vision Transformer for Real-Time UAV Tracking}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=8UqKVcrI7E},
}
Figure