Directional Deep Embedding and Appearance Learning for Fast Video Object Segmentation

Yingjie Yin, De Xu, Xingang Wang, Lei Zhang

Abstract

Semi-supervised Video Object Segmentation (VOS) typically faces a trade-off between accuracy and inference speed. While online fine-tuning methods achieve high accuracy, they are too slow for real-time applications. This paper introduces DDEAL, a framework that utilizes directional statistics in a spherical embedding space to achieve real-time VOS (25–31 fps) without fine-tuning. By leveraging cosine similarity and von Mises–Fisher (vMF) distributions, DDEAL robustly distinguishes targets from backgrounds with high computational efficiency.

Introduction

The goal of semi-supervised VOS is to segment a target object throughout a video sequence given its initial mask. Traditional high-performance methods rely on online fine-tuning, which involves updating the network on the first frame during inference. This process is computationally expensive, often requiring seconds per frame, making it unsuitable for real-time applications like robotics or interactive video editing.

Recent fine-tuning-free approaches attempt to use pixel-wise embedding learning in Euclidean space. However, these methods suffer from high computational complexity when calculating distances between millions of pixel pairs and the "curse of dimensionality," where probabilistic models struggle to define boundaries in high-dimensional Euclidean spaces.

Methodology

DDEAL addresses these challenges by shifting the matching process to a hyper-sphere, utilizing directional features.

Global Directional Matching Module (GDMM)

Instead of standard Euclidean distance, GDMM employs cosine similarity to match current frame pixels with the initial frame. Since cosine similarity between normalized vectors is equivalent to a dot product, this operation is implemented as highly optimized parallel convolutions. This provides stable "static cues" from the video's start.

Directional Appearance Model (DALM)

To capture "dynamic cues" as object appearance evolves, DDEAL uses a mixture of von Mises–Fisher (vMF) distributions—essentially the spherical equivalent of Gaussian distributions. A dual-pair strategy is employed: a "base" pair for general classification and a "supplementary" pair to specifically learn features that the base model misclassifies, enhancing robustness to appearance changes.

Overview of the DDEAL architecture, highlighting the Global Directional Matching and Directional Appearance Learning modules.

Experimental Results

The model was evaluated on DAVIS 2016, DAVIS 2017, and YouTube-VOS datasets.

Performance and Efficiency

DDEAL achieves a significant speedup over existing methods. It runs at 25–31 fps on a single GPU, making it approximately 325 times faster than fine-tuning-based methods like OnAVOS.

Accuracy (J&F Mean) vs. Inference Time. DDEAL variants (red star and square) achieve state-of-the-art speed while maintaining high accuracy.

On the DAVIS 2017 benchmark, DDEAL achieved a J&F mean score of 74.8%, outperforming other fine-tuning-free methods such as FEELVOS (71.5%) and A-GAME (70.0%). On the large-scale YouTube-VOS dataset, it reached an overall score of 71.3%, setting a new state-of-the-art for fine-tuning-free models.

Key Insights

  • Spherical Embedding: Normalizing features to unit length mitigates the curse of dimensionality, making directional features more discriminative for visual tasks than raw magnitude-based features.
  • Convolutions as Matchers: Treating first-frame features as convolutional kernels allows hardware-accelerated matching in a single pass.
  • Efficient Updates: The appearance model is updated via a running average of vMF parameters, avoiding slow backpropagation during inference and allowing the model to "learn" new appearances on the fly with almost zero overhead.

Conclusion

DDEAL demonstrates that directional embedding and appearance learning can provide a robust and extremely fast solution for VOS. While the model excels in real-time scenarios, it faces challenges with very thin objects (e.g., kite lines) due to feature blurring in convolutional operations. Future work will explore multi-scale features to better preserve fine-grained details without sacrificing the real-time speed that makes DDEAL so promising.

Summary updated

BibTeX
@article{yin2022directional,
  author = {Yingjie Yin and De Xu and Xingang Wang and Lei Zhang},
  title = {{Directional Deep Embedding and Appearance Learning for Fast Video Object Segmentation}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2022},
  doi = {10.1109/tnnls.2021.3054769},
}
Figure