URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark

Seonguk Seo, Joon-Young Lee, Bohyung HanECCV 2020 · OpenReview

Abstract

We propose URVOS, a unified referring video object segmentation network that estimates object masks in video frames based on natural language expressions. By jointly performing language-based segmentation and mask propagation using a single deep neural network with dual attention models, URVOS addresses the challenges of cross-modal alignment and temporal consistency. To support this research, we introduce Refer-Youtube-VOS, the first large-scale benchmark for this task. Our evaluations demonstrate that URVOS significantly outperforms existing methods on multiple benchmarks.

Introduction

Video Object Segmentation (VOS) traditionally relies on semi-supervised methods where a user provides a pixel-level mask for the initial frame. This process is labor-intensive and limits the scalability of VOS applications. Referring Video Object Segmentation (R-VOS) seeks to overcome this by using natural language queries (e.g., "the person in the red shirt") to identify the target object.

R-VOS presents two primary challenges: cross-modal alignment, requiring the model to map linguistic concepts to visual pixels, and temporal consistency, ensuring the mask tracks the correct object despite occlusions or appearance changes. Prior to this work, the lack of large-scale datasets and unified architectures hindered progress in this field.

The URVOS Architecture

URVOS is an end-to-end network designed to unify language-based segmentation and temporal mask propagation. Built upon a ResNet-50 backbone with a Feature Pyramid Network (FPN), the architecture incorporates two specialized attention mechanisms:

  1. Cross-Modal Attention Module: This module aligns linguistic features with visual features. By correlating every word in the query with spatial locations in the frame, it ensures the model grounds the segmentation in the provided description.
  2. Memory Attention Module: To maintain temporal consistency, this module leverages a memory bank of previous frames and their predicted masks. It allows the model to "remember" the target object's appearance and location over time.
URVOS successfully segments a dynamic object, such as a skateboarder, based on a language description.

The model employs a two-stage inference strategy. First, it identifies an "anchor" frame where the language-based segmentation is most confident. Second, it propagates the mask from this anchor throughout the video, utilizing both the language query and the memory of preceding frames to refine the segmentation.

Refer-Youtube-VOS Dataset

A major contribution of this work is the introduction of Refer-Youtube-VOS, the first large-scale dataset for R-VOS. It contains 3,975 videos and nearly 28,000 referring expressions, providing the necessary scale for training robust deep learning models. The dataset includes both "Full-video" descriptions and "First-frame" descriptions to account for different levels of temporal context.

Experimental Results

URVOS achieves state-of-the-art performance across several benchmarks. On the Refer-DAVIS17 dataset, it reached a J-score of 47.29, a significant improvement over previous methods. Ablation studies confirmed the necessity of the Memory Attention Module; removing it led to a substantial drop in performance, highlighting the importance of temporal context in video-level grounding.

The model maintains accuracy even in crowded environments with multiple potential targets, demonstrating robust cross-modal alignment.

Qualitative results show that URVOS is robust to challenging scenarios, including fast motion and complex backgrounds. The researchers also found that performance scales with dataset size, emphasizing the value of the Refer-Youtube-VOS benchmark.

Conclusion and Future Work

URVOS represents a significant step toward intuitive human-AI interaction in video analysis. By unifying language grounding and temporal tracking, it provides a robust framework for referring video object segmentation. Future research will focus on reducing the computational overhead of the memory mechanism and improving the model's ability to handle highly complex spatial relationships in language queries.

Summary updated

Abstract

We propose a unified referring video object segmentation network (URVOS). URVOS takes a video and a referring expression as inputs, and estimates the object masks referred by the given language expression in the whole video frames. Our algorithm addresses the challenging problem by performing language-based object segmentation and mask propagation jointly using a single deep neural network with a proper combination of two attention models. In addition, we construct the first large-scale referring video object segmentation dataset called Refer-Youtube-VOS. We evaluate our model on two benchmark datasets including ours and demonstrate the effectiveness of the proposed approach. The dataset is released at https://github.com/skynbe/Refer-Youtube-VOS .
BibTeX
@inproceedings{seo2020urvos,
  author = {Seonguk Seo and Joon-Young Lee and Bohyung Han},
  title = {{URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=MF96S43XkyG},
}
Figure