DeVos: Flow-Guided Deformable Transformer for Video Object Segmentation

Volodymyr Fedynyak, Yaroslav Romanus, Bohdan Hlovatskyi, Bohdan Sydor, Oles Dobosevych, Igor Babin, Roman RiazantsevWACV 2024

TL;DR

This paper introduces DeVOS, a novel Video Object Segmentation (VOS) architecture that significantly improves temporal consistency and robustness to appearance changes by combining memory-based matching with motion-guided deformable attention. DeVOS achieves state-of-the-art performance on key benchmarks while maintaining efficient inference speeds.

What Problem Does This Paper Address?

Video Object Segmentation (VOS) aims to accurately segment objects across video frames. While existing methods excel at matching features between frames, they often struggle with maintaining temporal consistency and handling significant appearance variations, scale changes, or complex motion. These challenges can lead to degraded segmentation masks over time. Previous approaches have tried to address this by introducing local matching or improving feature representations, but they often fail to effectively model complex scene dynamics and object transformations.

Methods

DeVOS introduces a novel architecture that tackles these challenges through several key innovations:

  • Adaptive Deformable Video Attention (ADVA): This is the core of DeVOS. ADVA is a deformable attention mechanism that adaptively samples key and value locations in the previous frame. It decouples motion and semantic information by using separate offset branches.
  • Motion-guided offsets: Optical flow is used to predict initial offsets, guiding the search region based on scene motion.
  • Semantic-guided offsets: Query features are used to predict local offsets, further refining the search region based on object semantics. This allows the attention mechanism to focus on relevant areas, making it robust to appearance and scale changes.

  • QK-flow Enhancement: To further leverage motion information, the authors integrate optical flow directly into the query and key feature maps. This helps distinguish between similar objects with different motion patterns and ensures stronger cycle consistency.

  • Multi-scale Matching: DeVOS utilizes features from multiple resolutions of the backbone encoder. This is crucial for capturing fine-grained details and handling overlapping objects with similar appearances, especially when combined with the adaptive attention.
  • Memory Bank and Long-Term Attention: Similar to other VOS methods, DeVOS employs a memory bank to store information from past frames. This memory is accessed via a long-term multi-scale attention block, complementing the short-term ADVA.

Datasets

  • Training: The model is trained on synthetic sequences derived from static image datasets, and then fine-tuned on DAVIS 2017, YouTube-VOS 2019, and optionally MOSE 2023.
  • Evaluation: The primary evaluation is performed on the DAVIS 2016, DAVIS 2017 (validation and test-dev), and YouTube-VOS 2019 (validation) benchmarks.

Key Insights and Ideas

  • Decoupling Motion and Semantics is Key: The paper argues that explicitly separating motion and semantic cues for offset prediction in deformable attention is crucial for robust VOS. This allows the model to better adapt to diverse scenarios.
  • Optical Flow as a Prior, Not Just a Feature: Instead of just using optical flow as an input feature, DeVOS leverages it to guide the search region for attention, acting as a strong prior for motion compensation.
  • Adaptive Attention for Robustness: The adaptive nature of ADVA, which dynamically adjusts the sampling locations based on both motion and semantic features, is a significant improvement over fixed window or global attention mechanisms.
  • Multi-scale Features Enhance Detail: Using features from multiple resolutions is essential for capturing fine-grained details, which is particularly important for segmenting small objects or objects with complex shapes.

Key Results

  • State-of-the-Art Performance: DeVOS achieves top-ranked performance on the DAVIS 2017 validation and test-dev sets (88.1% J&F and 83.0% J&F respectively) and YouTube-VOS 2019 validation (86.6% J&F).
  • Efficient Inference: The method maintains consistent and competitive run-time speeds, with FPS figures often exceeding those of other state-of-the-art methods. For example, DeVOS-L achieves 24.7 FPS on YouTube-VOS 2019.
  • Robustness to Challenges: Qualitative results demonstrate DeVOS's superior performance in handling complex shape and appearance changes, as well as rapid motion, as shown in examples like the gold-fish and mbike-trick sequences.

Limitations and Future Work

  • Dependence on Optical Flow Estimator: A practical limitation is the reliance on a pre-trained optical flow estimator. However, the authors note that their framework is flexible and can work with different flow estimation architectures without retraining.
  • ViT Backbone Impact: The paper briefly explores the impact of using a Swin-B backbone instead of ResNet50, showing a minor decrease in performance, suggesting further research is needed to fully understand the benefits of different backbones and pre-training strategies (like SAM).
  • Future Work: The authors suggest that their adaptive deformable video attention mechanism could be beneficial for other video-related tasks beyond VOS. Further research could also explore the benefits of large-scale pre-training of transformer encoders on instance segmentation for VOS.

Summary updated

BibTeX
@inproceedings{fedynyak2024devos,
  author = {Volodymyr Fedynyak and Yaroslav Romanus and Bohdan Hlovatskyi and Bohdan Sydor and Oles Dobosevych and Igor Babin and Roman Riazantsev},
  title = {{DeVos: Flow-Guided Deformable Transformer for Video Object Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure