Learning Tracking Representations from Single Point Annotations

Qiangqiang Wu, Antoni B. Chan

Abstract

Training deep visual trackers traditionally requires millions of expensive, hand-drawn bounding boxes. This paper introduces Soft Contrastive Learning (SoCL), a framework designed to learn high-quality tracking representations using only single-point annotations (clicking the object center). By leveraging objectness priors and soft-weighting mechanisms, SoCL achieves a 4.5x increase in annotation speed and reduces total costs by up to 85% while maintaining performance comparable to fully supervised, box-based baselines.

Introduction

Modern visual object tracking relies heavily on deep learning models trained on massive datasets such as GOT-10k or ILSVRC. The standard paradigm for supervising these models involves annotators drawing tight bounding boxes around targets in every frame. This process is labor-intensive, taking approximately 15 seconds per frame when including verification. While unsupervised methods attempt to bypass this cost, they often rely on noisy pseudo-labels that result in inferior tracking performance.

This research identifies a middle ground: point supervision. Clicking a target center is significantly faster—averaging just 2.27 seconds—but introduces two primary technical challenges: the missing scale (a point does not define object boundaries) and annotation noise (human clicks are rarely perfectly centered). The SoCL framework is proposed to bridge this gap, transforming imprecise point clicks into robust tracking representations.

The SoCL Framework

To overcome the limitations of point-based supervision, the authors propose a multi-stage approach that replaces rigid bounding boxes with probabilistic "soft" templates.

Target Objectness Prior (TOP) Map

To recover scale information without manual boxes, the system generates thousands of random and "EdgeBox" proposals around the annotated point. By aggregating objectness scores—derived from saliency and color contrast—the system creates a Target Objectness Prior (TOP) map. This map estimates the spatial extent of the target, providing a probabilistic guide for feature extraction.

Global and Local Soft Templates (GST & LST)

Instead of the traditional method of cropping a hard rectangular box, SoCL utilizes Global Soft Templates (GST). The model extracts features from the entire frame and computes a weighted sum based on the TOP map. This "soft" approach is inherently more resilient to imprecise clicks than a hard crop, as it avoids the binary inclusion/exclusion of pixels at the object boundaries. To handle occlusions, the model also employs Local Soft Templates (LST), which randomly sample subsets of the TOP map during training to ensure the representation remains robust even when the object is partially obscured.

Soft Negative Samples (SNS)

Contrastive learning typically requires a large memory bank of "hard negatives" (background regions that resemble the target). SoCL introduces Soft Negative Samples (SNS), which identifies high-similarity regions in the background of the current video on-the-fly. These regions are aggregated into negative templates, providing effective contrastive signals without the massive memory overhead typical of frameworks like MoCo.

Experimental Evaluation

The researchers evaluated SoCL across several major tracking benchmarks, including OTB-13, OTB-15, VOT-16, VOT-18, UAV123, LaSOT, and TrackingNet. For training, they utilized the GOT-10k dataset (1.4 million frames), "downgrading" existing bounding boxes to center points to simulate a low-cost annotation environment.

Performance and Efficiency

  • Cost-Effectiveness: When constrained to the same annotation time budget (e.g., 110 hours), a SoCL-trained SiamFC model significantly outperformed a standard box-trained model (56.7% vs. 52.0% AUC on OTB-13). This is primarily because the point-based approach allowed the model to ingest 4.5x more training data in the same timeframe.
  • Accuracy Parity: When trained on the full GOT-10k dataset, SoCL achieved results nearly identical to box-supervised versions, despite a 78% reduction in human labor time.
  • Noise Resilience: In stress tests involving 20 pixels of random noise added to point annotations, SoCL’s performance degraded by less than 1%, demonstrating superior robustness compared to brittle traditional methods.

Financial Impact

The authors estimate the cost of annotating GOT-10k with bounding boxes via Amazon Mechanical Turk at approximately $50,400. By utilizing the point-based SoCL framework, this cost drops to roughly $7,000, representing a massive reduction in the financial barrier to entry for developing high-performance trackers.

Conclusion and Future Work

SoCL demonstrates that "softness" in representation learning is a feature rather than a limitation. By using weighted probability maps instead of rigid boxes, models can become naturally robust to human error and learn effectively from minimal supervision.

While SoCL excels at learning discriminative representations, it does not entirely eliminate the need for boxes in all architectures. For state-of-the-art "Scale Regression" trackers (like TransT), a small amount of sparse box annotations (e.g., one box every 10 frames) was still required to train the specific network components responsible for predicting exact dimensions. Future research will focus on purely point-based methods for estimating target scale during inference, potentially removing the need for bounding boxes from the training pipeline entirely.

Summary updated

Abstract

Existing deep trackers are typically trained with largescale video frames with annotated bounding boxes. However, these bounding boxes are expensive and time-consuming to annotate, in particular for large scale datasets. In this paper, we propose to learn tracking representations from single point annotations (i.e., 4.5x faster to annotate than the traditional bounding box) in a weakly supervised manner. Specifically, we propose a soft contrastive learning (SoCL) framework that incorporates target objectness prior into end-to-end contrastive learning. Our SoCL consists of adaptive positive and negative sample generation, which is memory-efficient and effective for learning tracking representations. We apply the learned representation of SoCL to visual tracking and show that our method can 1) achieve better performance than the fully supervised baseline trained with box annotations under the same annotation time cost; 2) achieve comparable performance of the fully supervised baseline by using the same number of training frames and meanwhile reducing annotation time cost by 78% and total fees by 85%; 3) be robust to annotation noise.
BibTeX
@inproceedings{wu2024learning,
  author = {Qiangqiang Wu and Antoni B. Chan},
  title = {{Learning Tracking Representations from Single Point Annotations}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
}
Figure