Abstract
Most video object segmentation approaches process objects separately. This incurs high computational cost when multiple objects exist. In this paper, we propose AGSS-VOS to segment multiple objects in one feed-forward path via instance-agnostic and instance-specific modules. Information from the two modules is fused via an attention-guided decoder to simultaneously segment all object instances in one path. The whole framework is end-to-end trainable with instance IoU loss. Experimental results on Youtube-VOS and DAVIS-2017 dataset demonstrate that AGSS-VOS achieves competitive results in terms of both accuracy and efficiency.
Introduction
Video Object Segmentation (VOS) involves tracking and masking specific objects across a video sequence, typically starting from a user-provided mask in the first frame. The primary challenge in multi-object VOS is scalability. Most traditional methods process objects sequentially; if a video contains five objects, the computation time quintuples. Other approaches rely on slow "online learning" (fine-tuning on the first frame) or memory-intensive pixel-wise distance calculations.
AGSS-VOS addresses these issues with a "single-shot" framework designed to segment multiple objects simultaneously within a single feed-forward pass. By combining a shared feature extractor with lightweight instance-specific attention modules, it achieves a massive speedup—up to 200x faster than some competing methods—while maintaining state-of-the-art accuracy on major benchmarks.
Methodology
The AGSS-VOS framework splits the segmentation task into three main components to balance global context with instance-specific detail:
- Instance-Agnostic Module (IAM): This module uses a deep Siamese encoder-decoder to extract high-level semantic features shared across all objects. It identifies general "object-ness" and scene context without distinguishing between specific instances.
- Instance-Specific Module (ISM): A lightweight branch that generates unique attention maps for each individual object. It utilizes the previous frame's mask and optical flow to predict the expected location of specific objects in the current frame.
- Attention-Guided Decoder (AGD): This module fuses the shared features from the IAM with the specific attention maps from the ISM. By multiplying these inputs, the decoder "highlights" relevant features for a specific object while filtering out irrelevant data.
To resolve conflicts where a single pixel might be predicted for multiple objects, the model employs Probabilistic Normalization (a softmax-based aggregation). This ensures a definitive choice between overlapping instance predictions. Furthermore, the model is trained using an end-to-end Instance IoU Loss, which is more robust to variations in object size than standard cross-entropy.
Experimental Evaluation
The model was evaluated on the YouTube-VOS and DAVIS-2017 benchmarks. A key insight from the ablation studies was the importance of optical flow; using flow to warp masks from the previous frame proved critical for handling fast-moving objects and reflections.
Performance and Scalability
AGSS-VOS is remarkably efficient, clocking in at approximately 0.08 seconds per frame (12.5 FPS). As shown in the performance analysis, while competing methods like RGMP see their processing time skyrocket as more objects are added, AGSS-VOS’s runtime remains nearly flat.
On the YouTube-VOS dataset, the model achieved an overall score of 71.3%, outperforming all other methods that do not utilize time-consuming online learning. This demonstrates that treating instance-specific information as a spatial attention filter allows for high accuracy without the linear computational scaling typical of multi-object systems.
Conclusion
While AGSS-VOS offers significant improvements in efficiency, it faces challenges during heavy or prolonged occlusions. Because the system relies heavily on the immediate previous frame, it can "lose the trail" if an object is hidden for several frames. Future work may involve incorporating longer-term temporal memory or re-identification modules to improve robustness in complex, crowded scenes.
Summary updated