Co-Inference Discriminative Tracking Through Multi-Task Siamese Network

Yan Chen, Ji‐Xiang Du, Bineng Zhong

Abstract

Visual tracking is fundamentally a matching problem without prior class-agnostic object information. While Siamese networks leverage large-scale offline data to learn similarity functions, they often lack the discriminative and adaptive power necessary for complex real-world scenarios. We propose the Multi-stage Co-Inference (MSCI) tracker, which decomposes tracking into three collaborative sub-tasks: classification, regression, and detection. By employing a multi-task Siamese network with shared representations and a novel co-inference fusion strategy, MSCI achieves high discriminative power and adaptivity. Extensive evaluations on OTB2013, OTB2015, and VOT2016 benchmarks demonstrate that MSCI provides state-of-the-art accuracy while maintaining real-time performance at 27 FPS.

Introduction

Visual object tracking remains a challenging task in computer vision due to factors such as illumination variation, occlusion, and fast motion. Existing trackers typically fall into two categories: online adaptive trackers, which are accurate but computationally intensive, and Siamese trackers, which are fast but often struggle with distractors due to a lack of online model updates. The MSCI framework aims to combine the speed of Siamese networks with the robustness of adaptive methods by utilizing a multi-task learning approach.

Methodology: The MSCI Framework

The core of our approach is a multi-task Siamese network that processes a previous target template and a current search patch through a shared VGG19 backbone. The network branches into three parallel components:

  1. Classifier: Estimates the probability of a candidate patch being the target, aiding in re-identification.
  2. Regressor: Predicts precise bounding box offsets for position and scale.
  3. Residual Learning-based Detector: Utilizes residual blocks to capture complex appearance changes, serving as a robust fallback mechanism.
Architecture of the MSCI tracker, showing the multi-task Siamese network and co-inference logic.

Co-Inference Strategy

To effectively combine these components, we implement a tiered co-inference logic based on the classifier's confidence. If the classifier is highly confident, the regressor's output is used directly. For medium confidence, the search area is re-sampled to find the optimal patch. In cases of low confidence, the residual learning-based detector is activated to handle significant appearance variations, such as blur or deformation.

Experimental Results

The MSCI tracker was trained on large-scale datasets including ALOV300+, NUS-PRO, and ILSVC-2015 VID. Evaluation on standard benchmarks yielded significant performance gains. On the OTB2013 benchmark, MSCI achieved a 27.7% improvement in precision and a 22.5% improvement in AUC compared to the GOTURN tracker. The system operates at approximately 27 FPS on an NVIDIA GTX-980 GPU, meeting real-time requirements.

Qualitative tracking result showing multiple candidate bounding boxes during the inference process.

Ablation studies confirmed that the synergy between the three sub-tasks is critical, as removing any component led to a noticeable decline in tracking performance.

Conclusion

The MSCI tracker introduces a robust multi-task Siamese architecture that balances stability and plasticity. By integrating classification, regression, and residual-based detection through a co-inference strategy, the model achieves superior tracking performance without the need for expensive online fine-tuning. Future work will focus on dynamic search window adjustment and improved noise filtering to handle extreme camera motion and visual corruption.

Summary updated

Abstract

In essence, visual tracking is a matching problem without any prior information about a class-agnostic object. By leveraging large scale off-line training data, recent trackers based on Siamese networks usually expect to pre-learn underlying similarity functions before a tracking task even begins. Consequently, they lack discriminative and adaptive powers. To address the issues, we propose a multi-stage co-inference tracker (named MSCI) via a multi-task Siamese network, in which a complicated tracking task is divided into three complementary sub-tasks (i.e., classification, regression and detection). Firstly, we design a novel multi-task loss function to end-to-end train the multi-task Siamese network via jointly learning from three sub-tasks. The multi-task Siamese network contains three parallel yet collaborative output layers, which correspond to three key components of our tracker (i.e., classifier, regressor and residual learning based detector). By sharing representations within the components, we not only improve each component's generalization performance, but also enhance our tracker's discriminative power. Then, we design a co-inference approach to effectively fuse the complementary components. As a result, our tracker can avoid the pitfalls of purely single components and get reliably observations to improve its adaptive power. Comprehensive experiments on OTB2013, OTB2015 and VOT2016 validate the effectiveness and robustness of our MSCI tracker.
BibTeX
@inproceedings{chen2020co,
  author = {Yan Chen and Ji‐Xiang Du and Bineng Zhong},
  title = {{Co-Inference Discriminative Tracking Through Multi-Task Siamese Network}},
  booktitle = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3045036},
}
Figure