Abstract
Many recent methods for semi-supervised Video Object Segmentation (VOS) have achieved good performance by exploiting the annotated first frame via one-shot fine-tuning or mask propagation. However, heavily relying on the first frame may weaken the robustness for VOS, since video objects can show large variations through time. In this work, we propose a Dynamic Identity Propagation Network (DIPNet) that adaptively propagates and accurately segments the video objects over time. To achieve this, DIPNet factors the VOS task at each time step into a dynamic propagation phase and a spatial segmentation phase. The former utilizes a novel identity representation to adaptively propagate objects’ reference information over time, which enhances the robustness to videos’ temporal variations. The segmentation phase uses the propagated information to tackle the object segmentation as an easier static image problem that can be optimized via light-weight fine-tuning on the first frame, thus reducing the computational cost. As a result, by optimizing these two components to complement each other, we can achieve a robust system for VOS. Evaluations on four benchmark datasets show that DIPNet provides state-of-the-art performance with time efficiency.
Introduction
Semi-supervised Video Object Segmentation (VOS) requires tracking and segmenting a specific object throughout a video sequence, given only its initial mask in the first frame. This task is challenging due to temporal variations such as rotations, occlusions, and lighting changes. Traditionally, researchers have balanced a trade-off between slow, accurate "one-shot fine-tuning" methods (like OSVOS) and fast but drift-prone "mask propagation" methods (like RGMP). DIPNet, presented at WACV 2020, addresses this by decoupling the identity of the object ("who") from its spatial refinement ("where").
Methods: The Two-Phase Approach
DIPNet decomposes the VOS task into two distinct phases: Dynamic Identity Propagation and Spatial Instance Segmentation.
1. Dynamic Identity Propagation
Instead of relying on pixel-level templates, DIPNet utilizes an Identity Attention vector to track objects. This approach leverages the observation that specific CNN channels encode semantic object types.
- Identity Encoding: A ResNet-50 backbone extracts high-level features, and the target object's mask determines which feature channels are most relevant.
- Dynamic Updating: To handle appearance changes, the identity vector is updated at each frame using a weighted moving average: $$\alpha_t = \omega \cdot \beta_{t-1} + (1 - \omega) \cdot \alpha_{t-1}$$ This allows the model to adapt to recent changes while maintaining a memory of the object's historical appearance.
2. Spatial Instance Segmentation
Once the identity is propagated, the problem is treated as a static image segmentation task. The propagation phase generates a coarse 32x32 mask, which a Spatial Segmentation Network (based on a Cascaded Refinement Network) then refines into a pixel-perfect 512x512 segmentation using the original high-resolution frame. This refinement step is optimized via light-weight fine-tuning on the first frame, ensuring high accuracy without the heavy computational cost of full-model fine-tuning.
Key Results
DIPNet was evaluated on major benchmarks including DAVIS2016, DAVIS2017, YouTube-Object, and SegtrackV2.
- Efficiency: DIPNet processes frames in approximately 1.09 seconds, making it roughly 15x faster than OnAVOS (15.57s) while maintaining competitive accuracy.
- Accuracy: On DAVIS2016, it achieved a Mean $J$ (region similarity) of 0.858. On the multi-object DAVIS2017 dataset, it significantly outperformed other propagation-based methods like FAVOS and OSNM.
- Stability: Qualitative results demonstrate that DIPNet remains stable over long sequences where other methods suffer from significant drift or failure as the object's appearance evolves.
Conclusion
DIPNet represents a significant advancement in VOS by disentangling high-level semantic tracking from low-level detail extraction. By using channel-wise attention as a dynamic memory, the model achieves a robust balance between speed and accuracy. While the method still benefits from light-weight fine-tuning, its ability to handle complex temporal variations makes it a powerful tool for real-world video analysis.
Summary updated