Abstract
Semi-supervised Video Object Segmentation (VOS) typically faces a trade-off between accuracy and inference speed. While online fine-tuning methods achieve high accuracy, they are too slow for real-time applications. This paper introduces DDEAL, a framework that utilizes directional statistics in a spherical embedding space to achieve real-time VOS (25–31 fps) without fine-tuning. By leveraging cosine similarity and von Mises–Fisher (vMF) distributions, DDEAL robustly distinguishes targets from backgrounds with high computational efficiency.
Introduction
The goal of semi-supervised VOS is to segment a target object throughout a video sequence given its initial mask. Traditional high-performance methods rely on online fine-tuning, which involves updating the network on the first frame during inference. This process is computationally expensive, often requiring seconds per frame, making it unsuitable for real-time applications like robotics or interactive video editing.
Recent fine-tuning-free approaches attempt to use pixel-wise embedding learning in Euclidean space. However, these methods suffer from high computational complexity when calculating distances between millions of pixel pairs and the "curse of dimensionality," where probabilistic models struggle to define boundaries in high-dimensional Euclidean spaces.
Methodology
DDEAL addresses these challenges by shifting the matching process to a hyper-sphere, utilizing directional features.
Global Directional Matching Module (GDMM)
Instead of standard Euclidean distance, GDMM employs cosine similarity to match current frame pixels with the initial frame. Since cosine similarity between normalized vectors is equivalent to a dot product, this operation is implemented as highly optimized parallel convolutions. This provides stable "static cues" from the video's start.
Directional Appearance Model (DALM)
To capture "dynamic cues" as object appearance evolves, DDEAL uses a mixture of von Mises–Fisher (vMF) distributions—essentially the spherical equivalent of Gaussian distributions. A dual-pair strategy is employed: a "base" pair for general classification and a "supplementary" pair to specifically learn features that the base model misclassifies, enhancing robustness to appearance changes.
Experimental Results
The model was evaluated on DAVIS 2016, DAVIS 2017, and YouTube-VOS datasets.
Performance and Efficiency
DDEAL achieves a significant speedup over existing methods. It runs at 25–31 fps on a single GPU, making it approximately 325 times faster than fine-tuning-based methods like OnAVOS.
On the DAVIS 2017 benchmark, DDEAL achieved a J&F mean score of 74.8%, outperforming other fine-tuning-free methods such as FEELVOS (71.5%) and A-GAME (70.0%). On the large-scale YouTube-VOS dataset, it reached an overall score of 71.3%, setting a new state-of-the-art for fine-tuning-free models.
Key Insights
- Spherical Embedding: Normalizing features to unit length mitigates the curse of dimensionality, making directional features more discriminative for visual tasks than raw magnitude-based features.
- Convolutions as Matchers: Treating first-frame features as convolutional kernels allows hardware-accelerated matching in a single pass.
- Efficient Updates: The appearance model is updated via a running average of vMF parameters, avoiding slow backpropagation during inference and allowing the model to "learn" new appearances on the fly with almost zero overhead.
Conclusion
DDEAL demonstrates that directional embedding and appearance learning can provide a robust and extremely fast solution for VOS. While the model excels in real-time scenarios, it faces challenges with very thin objects (e.g., kite lines) due to feature blurring in convolutional operations. Future work will explore multi-scale features to better preserve fine-grained details without sacrificing the real-time speed that makes DDEAL so promising.
Summary updated