E3SN: Efficient End-to-End Siamese Network for Video Object Segmentation

Meng Lan, Yipeng Zhang, Qinning Xu, Lefei Zhang

Introduction

Semi-supervised Video Object Segmentation (VOS) is the task of tracking and segmenting a specific object throughout a video sequence, given its initial location in the first frame. While recent Siamese-based models like SiamMask have achieved impressive speeds, they often rely on complex multi-stage training pipelines and underutilize low-level spatial features. E3SN (Efficient End-to-End Siamese Network) addresses these limitations by introducing a supervised sampling strategy that enables true end-to-end training and a multilevel feature aggregation module for high-precision segmentation.

The Training Gap in Siamese VOS

Traditional fast VOS models typically suffer from a "training gap." For instance, SiamMask requires a two-stage process: first training a base model for object localization and then fine-tuning a separate refinement model for mask generation. This separation is necessitated by the high memory cost of predicting masks for every potential object candidate. Consequently, researchers must manually tune hyperparameters for each stage, complicating the development process.

Comparison between traditional two-stage training and the proposed end-to-end E3SN approach.

Methodology: End-to-End Learning and Feature Fusion

E3SN introduces two primary innovations to streamline the VOS pipeline and improve accuracy:

Supervised Sampling Strategy

To overcome memory bottlenecks, E3SN employs a "sample distribution matrix." Instead of generating masks for all candidates, the model calculates the Intersection over Union (IoU) between candidates and the ground truth. It then selects a fixed 3:1 ratio of positive to negative samples for training. This targeted sampling drastically reduces memory overhead, allowing the entire network—from feature extraction to mask refinement—to be trained simultaneously.

Multilevel Feature Aggregation

High-level semantic features are excellent for object identification but often lack the spatial precision needed for crisp boundaries. E3SN extracts features from multiple stages of a ResNet-50 backbone (specifically conv2, conv3, and conv4). These features are cross-correlated and fused using learnable weights, ensuring the model leverages both "big picture" semantic information and fine-grained spatial details like color and texture.

The full E3SN architecture, highlighting the Siamese backbone, correlation maps, and the refinement pathway with multilevel feature aggregation.

Experimental Results

E3SN was evaluated on the industry-standard DAVIS2016 and DAVIS2017 benchmarks. It demonstrates a superior balance between precision and real-time performance:

  • Accuracy: On DAVIS2016, E3SN achieved a 73.0% $\mathcal{J}$ Mean and 69.3% $\mathcal{F}$ Mean, outperforming SiamMask (71.7% / 67.8%). On DAVIS2017, it reached 56.1% $\mathcal{J}$ Mean.
  • Speed: The model maintains a high-speed performance of 46 frames per second (FPS) on an NVIDIA TITAN RTX, making it suitable for real-time applications.
  • Efficiency: The end-to-end approach reduced total training time from 22 hours to 20 hours while eliminating the need for manual multi-stage intervention.

Conclusion

E3SN represents a significant step forward in making video object segmentation both faster and easier to train. By intelligently sampling training data and aggregating hierarchical features, it achieves state-of-the-art performance with a simplified pipeline. While the model may still face challenges with long-term drifting in highly complex scenes, its ability to be initialized with a simple bounding box makes it a practical tool for real-world human-computer interaction.

Summary updated

Abstract

In the semi-supervised video object segmentation (VOS) field, SiamMask has achieved competitive accuracy and the fastest running speed. However, the two-stage training procedure requires additional manual intervention, and using only single-level features does not maximize the rich hierarchical feature information. This paper proposes an efficient end-to-end Siamese network for VOS. In particular, a supervised sampling strategy is designed to optimize the training procedure. Such an optimization facilitates the training of the entire model in an end-to-end manner. Moreover, a multilevel feature aggregation module is developed to enhance feature representability and improve segmentation accuracy. Experimental results on DAVIS2016 and DAVIS2017 datasets show that the proposed approach outperforms the SiamMask in accuracy with similar FPS. Moreover, this approach also achieves good accuracy-speed trade-off compared with that of other state-of-the-art VOS algorithms.
BibTeX
@inproceedings{lan2020e,
  author = {Meng Lan and Yipeng Zhang and Qinning Xu and Lefei Zhang},
  title = {{E3SN: Efficient End-to-End Siamese Network for Video Object Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/98},
}
Figure