Model Uncertainty Guides Visual Object Tracking

Lijun Zhou, Antoine Ledent, Qintao Hu, Ting Liu, Jianlin Zhang, Marius Kloft

Abstract

Visual object tracking requires continuous model updates to adapt to target appearance changes. However, limited memory often forces trackers to rely on naive sample selection strategies, which are prone to redundancy and noise from occlusions or blur. This paper introduces UATracker, a novel framework that leverages model uncertainty to intelligently select significant training samples. By filtering unreliable data and employing mixed-feature data augmentation, UATracker achieves state-of-the-art performance on major benchmarks, including OTB100 and VOT2018, while maintaining real-time speeds of 45 FPS.

Introduction

Visual object tracking involves following a specific target across a video sequence. Modern trackers utilize "online learning" to update their internal models as the video progresses. Despite its benefits, online learning faces a critical bottleneck: memory constraints. Trackers can typically only store a small subset of frames for training.

Existing methods often employ a naive selection strategy, prioritizing the initial frame and the most recent ones. This approach leads to two primary issues: redundancy, where temporally adjacent frames provide little new information, and noise, where the inclusion of blurred or occluded frames causes "tracking drift." UATracker addresses these challenges by using the model's own uncertainty to guide sample selection.

Methods

The UATracker architecture integrates uncertainty estimation directly into the tracking pipeline to improve the quality of the online training set.

The UATracker system architecture, illustrating the flow from feature extraction to Significant Sample Selection (S3) and online model prediction.

Significant Sample Selection (S3)

The core innovation is the Significant Sample Selection (S3) module. The tracker's regression branch predicts both the target's location and an estimate of aleatoric uncertainty (inherent data noise). The video sequence is divided into temporal clusters, and S3 calculates an "uncertainty-adjusted center" for each.

The S3 logic: The left panel shows the exclusion of a 'Wrong Sample' (noisy/occluded), while the right panel shows the selection of representative samples from temporal clusters.

As shown in the figure above, S3 filters out "garbage" samples—such as those with high uncertainty due to occlusion—and selects the frame closest to the cluster center with a sufficiently high confidence score. This ensures the training set remains diverse and representative of the target's true appearance over time.

Mixed-feature Data Augmentation and DCN

To further enhance robustness, the authors propose Mixed-feature Data Augmentation. Unlike standard pixel-level augmentation, this technique blends feature maps from different frames during offline training to simulate complex scenarios like partial occlusions. Additionally, the backbone (ResNet50) is enhanced with Deformable Convolutions (DCN), allowing the model to adapt its receptive field to objects undergoing rotation or scale changes.

Key Results and Discussion

UATracker was evaluated on major benchmarks including LaSOT, UAV123, OTB100, and VOT2018. It consistently outperformed the previous state-of-the-art, DiMP50:

  • OTB100: Achieved an AUC of 70.9%, a 2.5% improvement over DiMP50. On sequences specifically tagged with "occlusion," the performance gain rose to +5.2%, validating the effectiveness of the S3 strategy.
  • VOT2018: Reached a leading Expected Average Overlap (EAO) score of 0.458.
  • Efficiency: The tracker operates at 45 FPS on a standard GTX 1080ti, meeting the requirements for real-time application.

Conclusion

The primary insight of this work is that a model’s own "doubt" can serve as a powerful quality control mechanism. By using uncertainty as a filter, UATracker maintains a high-quality temporal memory of the target. Future work may involve more sophisticated, content-aware clustering logic to further reduce redundancy and dynamic thresholding to adapt to varying video difficulties.

Summary updated

Abstract

Model object trackers largely rely on the online learning of a discriminative classifier from potentially diverse sample frames. However, noisy or insufficient amounts of samples can deteriorate the classifiers' performance and cause tracking drift. Furthermore, alterations such as occlusion and blurring can cause the target to be lost. In this paper, we make several improvements aimed at tackling uncertainty and improving robustness in object tracking. Our first and most important contribution is to propose a sampling method for the online learning of object trackers based on uncertainty adjustment: our method effectively selects representative sample frames to feed the discriminative branch of the tracker, while filtering out noise samples. Furthermore, to improve the robustness of the tracker to various challenging scenarios, we propose a novel data augmentation procedure, together with a specific improved backbone architecture. All our improvements fit together in one model, which we refer to as the Uncertainty Adjusted Tracker (UATracker), and can be trained in a joint and end-to-end fashion. Experiments on the LaSOT, UAV123, OTB100 and VOT2018 benchmarks demonstrate that our UATracker outperforms state-of-the-art real-time trackers by significant margins.
BibTeX
@inproceedings{zhou2021model,
  author = {Lijun Zhou and Antoine Ledent and Qintao Hu and Ting Liu and Jianlin Zhang and Marius Kloft},
  title = {{Model Uncertainty Guides Visual Object Tracking}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16473},
}
Figure