Abstract: As a milestone for video object segmentation, one-shot video object segmentation (OSVOS) has achieved a large margin compared to the conventional optical-flow based methods regarding to the segmentation accuracy. Its excellent performance mainly benefit from the three-step training mechanism, that are: (1) acquiring object features on the base dataset (i.e. ImageNet), (2) training the parent network on the training set of the target dataset (i.e. DAVIS-2016) to be capable of differentiating the object of interest from the background. (3) online fine-tuning the interested object on the first frame of the target test set to overfit its appearance, then the model can be utilized to segment the same object in the rest frames of that video. In this paper, we argue that for the step (2), OSVOS has the limitation to 'overemphasize' the generic semantic object information while 'dilute' the instance cues of the object(s), which largely block the whole training process. Through adding a common module, video loss, which we formulate with various forms of constraints (including weighted BCE loss, high-dimensional triplet loss, as well as a novel mixed instance-aware video loss), to train the parent network in the step (2), the network is then better prepared for the step (3), i.e. online fine-tuning on the target instance. Through extensive experiments using different network structures as the backbone, we show that the proposed video loss module can improve the segmentation performance significantly, compared to that of OSVOS. Meanwhile, since video loss is a common module, it can be generalized to other fine-tuning based methods and similar vision tasks such as depth estimation and saliency detection.
Introduction
Video Object Segmentation (VOS) involves tracking and masking an object throughout a video, usually starting from a single annotated frame. A major milestone in this field is OSVOS, which utilizes a three-step training pipeline: pre-training on ImageNet, training a "parent network" on a VOS dataset to learn generic "objectness," and finally online fine-tuning on the specific object in a test video.
However, the standard parent network training (Step 2) often overemphasizes generic semantic information at the expense of unique instance cues. This "dilution" of instance-specific features makes the final fine-tuning step more difficult and often results in the model mistakenly segmenting background objects that share similar semantic categories with the target.
Methodology
To address this, the authors introduce a "Video Loss" (VL) module that acts as a regularizer during the parent network training phase. This module forces the network to distinguish between specific object instances rather than just learning a broad category of "objects."
Three variations of the Video Loss were explored:
- Two-Dimensional Video Loss (2D-VL): Uses unique Video IDs as labels to differentiate between instances of the same class across different videos.
- High-Dimensional Video Loss (HD-VL): Employs a triplet loss to map pixels into an embedding space where pixels from the same object are pulled together and pixels from different objects or background are pushed apart.
- Mixed Instance-aware Video Loss: Combines triplet loss with a Contrastive Center Loss to penalize distances between foreground and background cluster centers.
Crucially, this module is only used during training and does not add computational overhead to the final inference process.
Key Results
The proposed video loss consistently outperformed the baseline OSVOS across various architectures on the DAVIS-2016 dataset. Quantitatively, using a VGG16 backbone, the 2D-VL improved the J-Mean from 75.0 to 76.2. With a MobileNet backbone, the "Mixed" loss reached 67.5, compared to the 65.7 baseline.
Qualitatively, the method is significantly better at "cleaning up" segmentations. In sequences like camel or dog, where the original OSVOS would pick up distracting background elements, the VL-enhanced model successfully ignored the noise and maintained focus on the target instance. The video loss provided a clear performance boost in 12 out of 20 test sequences, particularly those featuring abrupt motion or similar-looking background objects.
Conclusion
By refining the pre-training phase to include instance-aware constraints, this work demonstrates that the fine-tuning paradigm of OSVOS still holds significant potential. While the online fine-tuning step remains computationally expensive, the improved parent network provides a much stronger foundation for one-shot learning, leading to more robust and accurate video object segmentation.
Summary updated