Abstract
Visual tracking is fundamentally a matching problem without prior class-agnostic object information. While Siamese networks leverage large-scale offline data to learn similarity functions, they often lack the discriminative and adaptive power necessary for complex real-world scenarios. We propose the Multi-stage Co-Inference (MSCI) tracker, which decomposes tracking into three collaborative sub-tasks: classification, regression, and detection. By employing a multi-task Siamese network with shared representations and a novel co-inference fusion strategy, MSCI achieves high discriminative power and adaptivity. Extensive evaluations on OTB2013, OTB2015, and VOT2016 benchmarks demonstrate that MSCI provides state-of-the-art accuracy while maintaining real-time performance at 27 FPS.
Introduction
Visual object tracking remains a challenging task in computer vision due to factors such as illumination variation, occlusion, and fast motion. Existing trackers typically fall into two categories: online adaptive trackers, which are accurate but computationally intensive, and Siamese trackers, which are fast but often struggle with distractors due to a lack of online model updates. The MSCI framework aims to combine the speed of Siamese networks with the robustness of adaptive methods by utilizing a multi-task learning approach.
Methodology: The MSCI Framework
The core of our approach is a multi-task Siamese network that processes a previous target template and a current search patch through a shared VGG19 backbone. The network branches into three parallel components:
- Classifier: Estimates the probability of a candidate patch being the target, aiding in re-identification.
- Regressor: Predicts precise bounding box offsets for position and scale.
- Residual Learning-based Detector: Utilizes residual blocks to capture complex appearance changes, serving as a robust fallback mechanism.
Co-Inference Strategy
To effectively combine these components, we implement a tiered co-inference logic based on the classifier's confidence. If the classifier is highly confident, the regressor's output is used directly. For medium confidence, the search area is re-sampled to find the optimal patch. In cases of low confidence, the residual learning-based detector is activated to handle significant appearance variations, such as blur or deformation.
Experimental Results
The MSCI tracker was trained on large-scale datasets including ALOV300+, NUS-PRO, and ILSVC-2015 VID. Evaluation on standard benchmarks yielded significant performance gains. On the OTB2013 benchmark, MSCI achieved a 27.7% improvement in precision and a 22.5% improvement in AUC compared to the GOTURN tracker. The system operates at approximately 27 FPS on an NVIDIA GTX-980 GPU, meeting real-time requirements.
Ablation studies confirmed that the synergy between the three sub-tasks is critical, as removing any component led to a noticeable decline in tracking performance.
Conclusion
The MSCI tracker introduces a robust multi-task Siamese architecture that balances stability and plasticity. By integrating classification, regression, and residual-based detection through a co-inference strategy, the model achieves superior tracking performance without the need for expensive online fine-tuning. Future work will focus on dynamic search window adjustment and improved noise filtering to handle extreme camera motion and visual corruption.
Summary updated