Abstract
Target tracking based on Siamese networks has reached state-of-the-art performance but remains limited in semantic feature extraction. This paper proposes FPSiamRPN, a novel method to distinguish positive and negative samples by fusing feature maps from different layers of a deep neural network backbone and feeding them into a Region Proposal Network (RPN). Additionally, a new loss term is introduced to achieve self-adjusting learning of discriminative embedding features for target objects with similar semantics. In the tracking stage, one-shot detection is utilized, fixing the first frame as the tracking weight for subsequent frames. Our method achieves outstanding performance on several benchmark datasets, including OTB2015, VOT2016, VOT2018, and VOT2019.
Introduction
Siamese networks have become a dominant force in visual object tracking by treating the task as a similarity learning problem: determining how much a candidate region in the current frame resembles the initial template. However, standard Siamese trackers face two major hurdles. First, they often rely on shallow features or a single deep layer, creating a trade-off where deep layers provide high-level semantics but lose the fine-grained spatial details needed for precise localization. Second, they struggle with "distractors"—objects belonging to the same category as the target (e.g., one specific runner in a marathon)—which often causes the tracker to drift.
Methods
FPSiamRPN (Feature Pyramid Siamese Region Proposal Network) addresses these challenges through three main architectural innovations:
- Feature Pyramid Fusion: Instead of using only the final layer of a backbone, the model utilizes a modified ResNet50 and fuses feature maps from different stages (conv2 through conv5). This allows the network to leverage both high-level semantic information and low-level structural details simultaneously.
- Novel Instance Loss ($L_{inst}$): To mitigate the distractor problem, a new loss term is introduced. During training, the model uses a softmax-based score to compare the target's score against all other anchor proposals, forcing the model to learn a more discriminative embedding space where the target stands out among similar-looking objects.
- Depth-wise Cross-Correlation: The model employs depth-wise cross-correlation to combine the template and search branches. This approach is more computationally efficient than traditional cross-correlation and helps maintain a balanced parameter distribution, leading to more stable training.
Key Insights and Results
The core innovation of FPSiamRPN is the "zoom" effect provided by the feature pyramid. By looking at the target through multiple layers at once, the deep layers confirm the object's category, while the shallower layers confirm specific identifying details. This is visualized through feature maps that highlight different levels of abstraction across the network layers.
Performance Benchmarks
The model was trained on massive datasets including COCO, ImageNet DET, and ImageNet VID. Evaluation on gold-standard benchmarks demonstrated superior performance:
- VOT2016: Achieved an Expected Average Overlap (EAO) of 0.354, ranking 1st among compared methods like C-COT and SiamRPN.
- OTB2015: Precision improved by 5.3% compared to DaSiamRPN, showing superior accuracy in pinpointing the target center.
- VOT2018: Improved EAO by 2.9% over the baseline SiamRPN.
- Efficiency: Despite the complex feature fusion, the model maintains a high speed (32.2 EFO), making it suitable for real-time applications.
Conclusion and Future Work
While FPSiamRPN performs excellently on standard benchmarks, challenges remain in the VOT2019 dataset, particularly in scenes with extreme motion blur or where identical targets cross paths. Future work will focus on improving the model's robustness in these "failed scenes" by incorporating temporal information or more advanced box refinement procedures to handle inaccurate candidate proposals.
Summary updated