Updatable Siamese Tracker with Two-stage One-shot Learning

Xinglong Sun, Guangliang Han, Lihong Guo, Tingfa Xu, Jianan Li, Peixun Liu

This paper, titled "Updatable Siamese Tracker with Two-stage One-shot Learning," presents a significant step forward in making Siamese-based visual object trackers more robust to real-world changes. By re-imagining the tracking process through the lens of meta-learning, the authors introduce a tracker that can "learn how to update itself" on the fly.

TL;DR

SiamTOL is an updatable Siamese tracker that uses a novel two-stage one-shot learning framework to integrate new object features online. By combining the initial ground-truth template with the latest reliable tracking results via a residual fusion module, it achieves state-of-the-art accuracy on major benchmarks while maintaining a high real-time speed of 75 FPS.

What Problem Does This Paper Address?

Siamese networks have dominated visual object tracking because they are incredibly efficient. They treat tracking as a simple similarity matching problem: "Find the patch in the current frame that looks most like the object in the first frame."

However, this "first-frame-only" approach is a double-edged sword. While fast, these trackers are often "blind" to appearance changes. If an object rotates, moves into different lighting, or undergoes deformation, the initial template becomes obsolete, leading to tracking failure. Existing solutions—like online fine-tuning—are often too slow for real-time use or are structurally separate from the tracker, meaning the update mechanism and the tracking mechanism aren't optimized to work together.

Methods

The authors propose SiamTOL (Updatable Siamese Tracker with Two-stage One-shot Learning). The core innovation is treating the Siamese network not just as a matcher, but as a one-shot learner that can be extended to two stages.

1. Two-stage One-shot Learner

In a standard Siamese network, the exemplar branch acts as a "learnet" that predicts the weights for a classifier used in the detection branch. SiamTOL extends this by adding an Update Branch. As shown in the framework overview below, the network now processes three inputs: the initial exemplar, a recent reliable tracking result (the update sample), and the current search region.

Figure 2. Overview of the proposed framework, consisting of the shared backbone network, feature fusion network and region proposal network. The backbone networks in both exemplar branch and update branch as well as the feature fusion network works as a two-stage learnet, which has an ability to generate the part weights of the RPN block in classifier (detection branch).

2. Residual Feature Fusion

Instead of simply replacing the old template with a new one, SiamTOL uses a Feature Fusion Network (FFN) based on residual learning. It calculates a "residual" from the latest object features and adds it to the initial template. This ensures the tracker stays anchored to the reliable ground truth of the first frame while adapting to recent variations.

3. Multi-aspect Loss Training

To prevent the model from overfitting or becoming overly reliant on potentially noisy update samples, the authors designed a multi-aspect loss function. It combines three distinct losses:

  • Basic Loss: Matching the initial exemplar to the search region.
  • Update Loss: Matching the latest update sample to the search region.
  • Overall Loss: Matching the fused template to the search region.
Figure 3. Flowchart of the proposed Multi-aspect loss training.

Datasets

The model was trained on a massive collection of datasets, including ImageNet VID, YouTube-BB, ImageNet DET, COCO, LaSOT, and GOT-10k.

For evaluation, the authors tested SiamTOL against six major benchmarks: OTB100, VOT2018, VOT2019, LaSOT, UAV123, and GOT-10k. These datasets cover a wide range of challenges, from long-term tracking to aerial views and complex background clutter.

Key Insights and Ideas

  • Siamese as Meta-Learning: The most elegant idea here is reinterpreting the exemplar branch as a meta-learner. This allows the update mechanism to be integrated directly into the architecture rather than being an "add-on."
  • The Power of Residuals: The choice of residual learning for fusion is critical. It acknowledges that the first frame is the only "pure" ground truth. By treating updates as residuals, the tracker avoids "noise pollution" that often causes other online updaters to drift away from the target.
  • Efficiency is Key: Unlike methods that require heavy online optimization (like SGD during inference), SiamTOL updates via a simple feed-forward pass through the FFN, keeping it well above the 30-60 FPS threshold required for most real-time applications.

Key Results

SiamTOL demonstrates a superior balance of quality and speed compared to other state-of-the-art (SOTA) trackers.

  • VOT2018: Achieved a leading EAO (Expected Averaged Overlap) of 0.459, outperforming high-profile trackers like SiamBAN (0.452) and DiMP (0.440).
  • Speed: It runs at 75 FPS on an NVIDIA Titan XP. As seen in Figure 5, it sits in the "sweet spot" of the top-right corner, offering higher quality than the baseline SiamRPN++ with only a minor trade-off in speed.
  • Robustness: On the VOT2019 dataset, SiamTOL showed a significant 10% improvement in robustness over the baseline, proving that the online update effectively prevents tracking drift.
Figure 5. The comparison of the quality and the speed of some state-of-the-art trackers on VOT2018. The baseline is the simpli- fied SiamRPN++ composed by our utilized backbone and RPN networks

Limitations and Future Work

While SiamTOL is robust, it still relies on a confidence threshold ($T_m = 0.9$) to select update samples. If the tracker is already in a state of failure but produces a high-confidence (yet incorrect) prediction, the update could potentially reinforce the error.

The authors suggest that future work could further explore how to handle "noise pollution" in update samples more aggressively. Additionally, while the current fusion uses a compact ResNet-50 backbone, exploring even deeper or more specialized backbones could yield further gains in complex long-term tracking scenarios.

Summary updated

Abstract

Offline Siamese networks have achieved very promising tracking performance, especially in accuracy and efficiency. However, they often fail to track an object in complex scenes due to the incapacity in online update. Traditional updaters are difficult to process the irregular variations and sampling noises of objects, so it is quite risky to adopt them to update Siamese networks. In this paper, we first present a two-stage one-shot learner, which can predict the local parameters of primary classifier with object samples from diverse stages. Then, an updatable Siamese network is proposed based on the learner (SiamTOL), which is able to complement online update by itself. Concretely, we introduce an extra inputting branch to sequentially capture the latest object features, and design a residual module to update the initial exemplar using these features. Besides, an effective multi-aspect training loss is designed for our network to avoid overfit. Extensive experimental results on several popular benchmarks including OTB100, VOT2018, VOT2019, LaSOT, UAV123 and GOT10k manifest that the proposed tracker achieves the leading performance and outperforms other state-of-the-art methods
BibTeX
@article{sun2021updatable,
  author = {Xinglong Sun and Guangliang Han and Lihong Guo and Tingfa Xu and Jianan Li and Peixun Liu},
  title = {{Updatable Siamese Tracker with Two-stage One-shot Learning}},
  journal = {Pattern Recognition},
  year = {2021},
  doi = {10.1016/j.patcog.2023.109965},
}
Figure