FPSiamRPN: Feature Pyramid Siamese Network With Region Proposal Network for Target Tracking

Yunbo Rao, Yiming Cheng, Junmin Xue, Jiansu Pu, Qiujie Wang, Rize Jin, Qifei WangIEEE Access 2020 · OpenReview

Abstract

Target tracking based on Siamese networks has reached state-of-the-art performance but remains limited in semantic feature extraction. This paper proposes FPSiamRPN, a novel method to distinguish positive and negative samples by fusing feature maps from different layers of a deep neural network backbone and feeding them into a Region Proposal Network (RPN). Additionally, a new loss term is introduced to achieve self-adjusting learning of discriminative embedding features for target objects with similar semantics. In the tracking stage, one-shot detection is utilized, fixing the first frame as the tracking weight for subsequent frames. Our method achieves outstanding performance on several benchmark datasets, including OTB2015, VOT2016, VOT2018, and VOT2019.

Introduction

Siamese networks have become a dominant force in visual object tracking by treating the task as a similarity learning problem: determining how much a candidate region in the current frame resembles the initial template. However, standard Siamese trackers face two major hurdles. First, they often rely on shallow features or a single deep layer, creating a trade-off where deep layers provide high-level semantics but lose the fine-grained spatial details needed for precise localization. Second, they struggle with "distractors"—objects belonging to the same category as the target (e.g., one specific runner in a marathon)—which often causes the tracker to drift.

Methods

FPSiamRPN (Feature Pyramid Siamese Region Proposal Network) addresses these challenges through three main architectural innovations:

  1. Feature Pyramid Fusion: Instead of using only the final layer of a backbone, the model utilizes a modified ResNet50 and fuses feature maps from different stages (conv2 through conv5). This allows the network to leverage both high-level semantic information and low-level structural details simultaneously.
  2. Novel Instance Loss ($L_{inst}$): To mitigate the distractor problem, a new loss term is introduced. During training, the model uses a softmax-based score to compare the target's score against all other anchor proposals, forcing the model to learn a more discriminative embedding space where the target stands out among similar-looking objects.
  3. Depth-wise Cross-Correlation: The model employs depth-wise cross-correlation to combine the template and search branches. This approach is more computationally efficient than traditional cross-correlation and helps maintain a balanced parameter distribution, leading to more stable training.
Architecture of the SiameseRPN module with depth-wise cross-correlation.

Key Insights and Results

The core innovation of FPSiamRPN is the "zoom" effect provided by the feature pyramid. By looking at the target through multiple layers at once, the deep layers confirm the object's category, while the shallower layers confirm specific identifying details. This is visualized through feature maps that highlight different levels of abstraction across the network layers.

Visualization of feature maps across different layers of the network.

Performance Benchmarks

The model was trained on massive datasets including COCO, ImageNet DET, and ImageNet VID. Evaluation on gold-standard benchmarks demonstrated superior performance:

  • VOT2016: Achieved an Expected Average Overlap (EAO) of 0.354, ranking 1st among compared methods like C-COT and SiamRPN.
  • OTB2015: Precision improved by 5.3% compared to DaSiamRPN, showing superior accuracy in pinpointing the target center.
  • VOT2018: Improved EAO by 2.9% over the baseline SiamRPN.
  • Efficiency: Despite the complex feature fusion, the model maintains a high speed (32.2 EFO), making it suitable for real-time applications.

Conclusion and Future Work

While FPSiamRPN performs excellently on standard benchmarks, challenges remain in the VOT2019 dataset, particularly in scenes with extreme motion blur or where identical targets cross paths. Future work will focus on improving the model's robustness in these "failed scenes" by incorporating temporal information or more advanced box refinement procedures to handle inaccurate candidate proposals.

Summary updated

Abstract

Target tracking based on Siamese network has reached the state-of-the-art performance. However is still limited in semantic feature extraction. In this paper, we propose a novel method to distinguish positive and negative samples. Taking deep neural network as the backbone, we fuse the feature maps from different layers and feed it to RPN (Region Proposal Network). In addition, we use a loss term for loss function to achieve self-adjusting and learn more discriminative embedding features of target objects with similar semantics. In the tracking stage, one-shot detection is used as the reference, fix the first frame as the weight of tracking to track the subsequent frames. Our method has achieved outstanding performance on several benchmark data set, such as: OTB2015, VOT2016, VOT2018, and VOT2019 et al.
BibTeX
@inproceedings{rao2020fpsiamrpn,
  author = {Yunbo Rao and Yiming Cheng and Junmin Xue and Jiansu Pu and Qiujie Wang and Rize Jin and Qifei Wang},
  title = {{FPSiamRPN: Feature Pyramid Siamese Network With Region Proposal Network for Target Tracking}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=SMjhaQkhjqD},
}
Figure