Abstract
In this paper, we propose a feature embedding based video object segmentation (VOS) method which is simple, fast and effective. The current VOS task involves two main challenges: object instance differentiation and cross-frame instance alignment. Most state-of-the-art matching based VOS methods simplify this task into a binary segmentation task and tackle each instance independently. In contrast, we decompose the VOS task into two subtasks: global embedding learning that segments foreground objects of each frame in a pixel-to-pixel manner, and instance feature embedding learning that separates instances. The outputs of these two subtasks are fused to obtain the final instance masks quickly and accurately. Through using the relation among different instances per-frame as well as temporal relation across different frames, the proposed network learns to differentiate multiple instances and associate them properly in one feed-forward manner. Extensive experimental results on the challenging DAVIS and Youtube-VOS datasets show that our method achieves better performances than most counterparts in each case.
Introduction: The Multi-Object Challenge
Semi-supervised video object segmentation (VOS) requires tracking and segmenting specific objects across a video sequence given their masks in the first frame. A significant limitation of current state-of-the-art "matching-based" methods, such as the Space-Time Memory (STM) network, is their treatment of multi-object scenarios. These models typically handle each object instance as an independent binary segmentation task. Consequently, if a scene contains $N$ objects, the model must perform $N$ separate forward passes.
This sequential approach leads to two primary issues:
- Computational Inefficiency: Inference time increases linearly with the number of objects, making real-time performance difficult as scene complexity grows.
- Lack of Inter-Instance Context: By isolating objects, the model ignores valuable spatial and temporal relationships between instances, such as relative positioning and occlusions.
The "Once-for-All" Framework
To address these limitations, we propose a framework that processes all object instances simultaneously in a single feed-forward pass. This "once-for-all" approach maintains high efficiency regardless of the number of instances while leveraging global scene context.
Dual-Branch Architecture
Our architecture decomposes the VOS task into two complementary subtasks: global saliency detection and instance-specific embedding.
- Global Feature Embedding: This branch focuses on "objectness" at a pixel level. It generates a high-resolution saliency map to distinguish all foreground objects from the background, ensuring sharp and accurate boundaries.
- Instance Feature Embedding: This branch learns to differentiate between specific individuals (e.g., distinguishing two different dogs). It uses a label encoder to map initial masks into a high-dimensional embedding space.
- Query-to-Memory with Temporal Attention: The model matches features from a "support set" (past frames) to the current "query" frame. A temporal attention module adaptively weights these support frames, typically prioritizing the high-quality first frame to prevent tracking drift.
The outputs from both branches are fused—often using a Hadamard product—and passed through a decoder to produce the final refined multi-instance masks.
Experimental Evaluation
The proposed method was evaluated on the DAVIS 2017 and YouTube-VOS 2018 datasets, which are standard benchmarks for the VOS task.
- Accuracy: On DAVIS 2017 validation, our method achieved a $\mathcal{J}\&\mathcal{F}$ score of 82.7, surpassing STM's 81.8. It also performed strongly on YouTube-VOS 2018 with an overall score of 80.6, demonstrating excellent generalization to unseen object classes.
- Efficiency: The full model operates at 6.6 FPS, while a "fast" variant reaches 9.1 FPS. Crucially, as shown in Figure 1, our method's speed remains relatively stable as the number of instances increases, unlike sequential methods where performance degrades linearly.
- Robustness: Qualitative analysis indicates that the global branch helps the model handle complex occlusions more effectively than independent-instance methods by maintaining a holistic understanding of the scene layout.
Conclusion
By shifting from sequential binary tasks to a unified embedding-based framework, we achieve a more efficient and context-aware VOS solution. The decomposition into global and instance branches allows for both high-resolution boundary accuracy and robust instance differentiation, setting a new standard for multi-object video segmentation.
Summary updated