Online Siamese Network for Visual Object Tracking

Shuo Chang, Wei Li, Yifan Zhang, Zhiyong FengSensors 2019 · DOI

TL;DR

This paper introduces an "Online Siamese Network for Visual Object Tracking" (OSNV) that addresses the limitations of offline-trained Siamese networks in adapting to changing target appearances. By incorporating online weight updates and an improved loss function that combines cross-entropy and contrastive losses, OSNV demonstrates robust tracking performance across various challenging scenarios.

What Problem Does This Paper Address?

Visual object tracking is a critical task in many applications, but it's notoriously difficult due to challenges like occlusion, scale variations, illumination changes, and background clutter. Traditional offline-trained Siamese networks, while powerful for feature extraction, struggle when the target's appearance changes significantly during tracking because they cannot learn from new data. This inflexibility leads to performance degradation. The paper aims to overcome this by enabling the Siamese network to learn and adapt online during the tracking process, effectively incorporating instance-specific domain knowledge.

Methods

The core of the proposed method is an "Online Siamese Network" (OSNV). Unlike previous offline approaches, OSNV updates its network weights in an online manner during tracking. This allows it to adapt to appearance changes.

Key methodological contributions include:

  • Lightweight Siamese Network: A four-layer Siamese network is designed for efficient feature extraction. This network is trained using feature maps from VGG-M, and it's shown to effectively reduce redundant information compared to raw VGG-M features.
  • Improved Contrastive Loss: The paper proposes a novel loss function that integrates cross-entropy loss with the standard contrastive loss. This aims to leverage the strengths of both: cross-entropy for increasing inter-class variations (discriminating target from background) and contrastive loss for reducing intra-class variations (keeping features of the same target similar). This combined loss is designed to enhance discrimination capability.
  • Bayesian Verification Model: For candidate selection, the authors adapt a Bayesian verification model, inspired by face verification techniques. This model is used to compute a similarity score for candidate bounding boxes, helping to select the most relevant one.
  • Online Updating Strategy: The network is updated using tracking results and sample candidates collected during the tracking process, forming an online training set. This continuous learning is crucial for adapting to appearance changes.

Datasets

The paper evaluates its proposed algorithm on several standard visual object tracking benchmarks:

  • OTB-2013: Contains 51 test sequences.
  • OTB-50: A subset of OTB-2015 with 50 challenging sequences.
  • OTB-2015: A larger benchmark with 100 test sequences.
  • VOT-2016: A challenging benchmark for evaluating tracking robustness.
  • TempleColor: A dataset with 129 video sequences.

The paper does not introduce any new datasets.

Key Insights and Ideas

  • Online Learning is Crucial for Robustness: The most significant insight is that offline-trained models are inherently limited when target appearance changes. Online updating is essential for robust tracking in dynamic environments.
  • Loss Function Synergy: Combining cross-entropy and contrastive loss is more effective than using either alone. This hybrid approach balances the need to distinguish targets from backgrounds with the need to maintain consistent target representations.
  • Face Verification Techniques for Object Tracking: The successful application of a Bayesian verification model, originally developed for face verification, to object tracking suggests a promising avenue for leveraging techniques from related computer vision domains.
  • Feature Representation Matters: The paper demonstrates that a well-designed, lightweight Siamese network can produce more discriminative features than standard pre-trained network features (like VGG-M) for tracking tasks, especially after online adaptation.

Key Results

  • Performance Improvement: OSNV achieves comparable or superior performance to several state-of-the-art trackers on OTB benchmarks. For instance, it shows a 1.5% improvement over the logistic loss-based Siamese network (OSNV_Log) in ablation studies.
  • Robustness: OSNV demonstrates improved robustness to appearance changes compared to offline Siamese methods like SiamFC_3s and SINT_noflow, particularly in sequences with significant target deformation or changes.
  • Speed: The algorithm runs at an average of 1.2405 fps on a Nvidia TitanX GPU, making it suitable for real-time applications.
  • Ablation Study: The ablation study confirms the effectiveness of the proposed improved contrastive loss and the online updating mechanism. Removing these components leads to significant performance degradation.

Limitations and Future Work

  • Severe Deformation: The authors acknowledge that OSNV struggles with severe deformation and dramatic scale changes, where it can lose track of the target and has difficulty recovering.
  • Incorrect Knowledge: Due to online updating, the network can sometimes learn incorrect knowledge, hindering recovery from mistakes.
  • Future Work:
    • Designing more complex Siamese networks to handle severe deformation.
  • Exploring other candidate selection methods beyond Bayesian verification.
  • Investigating the suitability of loss functions for model training irrespective of neural network structures.
  • Further research into online model updating for Siamese networks to improve tracking performance.

Summary updated

Abstract

Offline-trained Siamese networks are not robust to the environmental complication in visual object tracking. Without online learning, the Siamese network cannot learn from instance domain knowledge and adapt to appearance changes of targets. In this paper, a new lightweight Siamese network is proposed for feature extraction. To cope with the dynamics of targets and backgrounds, the weight in the proposed Siamese network is updated in an online manner during the tracking process. In order to enhance the discrimination capability, the cross-entropy loss is integrated into the contrastive loss. Inspired by the face verification algorithm DeepID2, the Bayesian verification model is applied for candidate selection. In general, visual object tracking can benefit from face verification algorithms. Numerical results suggest that the newly developed algorithm achieves comparable performance in public benchmarks.
BibTeX
@inproceedings{chang2019online,
  author = {Shuo Chang and Wei Li and Yifan Zhang and Zhiyong Feng},
  title = {{Online Siamese Network for Visual Object Tracking}},
  booktitle = {Sensors},
  year = {2019},
  doi = {10.3390/s19081858},
}
Figure