Abstract
We propose URVOS, a unified referring video object segmentation network that estimates object masks in video frames based on natural language expressions. By jointly performing language-based segmentation and mask propagation using a single deep neural network with dual attention models, URVOS addresses the challenges of cross-modal alignment and temporal consistency. To support this research, we introduce Refer-Youtube-VOS, the first large-scale benchmark for this task. Our evaluations demonstrate that URVOS significantly outperforms existing methods on multiple benchmarks.
Introduction
Video Object Segmentation (VOS) traditionally relies on semi-supervised methods where a user provides a pixel-level mask for the initial frame. This process is labor-intensive and limits the scalability of VOS applications. Referring Video Object Segmentation (R-VOS) seeks to overcome this by using natural language queries (e.g., "the person in the red shirt") to identify the target object.
R-VOS presents two primary challenges: cross-modal alignment, requiring the model to map linguistic concepts to visual pixels, and temporal consistency, ensuring the mask tracks the correct object despite occlusions or appearance changes. Prior to this work, the lack of large-scale datasets and unified architectures hindered progress in this field.
The URVOS Architecture
URVOS is an end-to-end network designed to unify language-based segmentation and temporal mask propagation. Built upon a ResNet-50 backbone with a Feature Pyramid Network (FPN), the architecture incorporates two specialized attention mechanisms:
- Cross-Modal Attention Module: This module aligns linguistic features with visual features. By correlating every word in the query with spatial locations in the frame, it ensures the model grounds the segmentation in the provided description.
- Memory Attention Module: To maintain temporal consistency, this module leverages a memory bank of previous frames and their predicted masks. It allows the model to "remember" the target object's appearance and location over time.
The model employs a two-stage inference strategy. First, it identifies an "anchor" frame where the language-based segmentation is most confident. Second, it propagates the mask from this anchor throughout the video, utilizing both the language query and the memory of preceding frames to refine the segmentation.
Refer-Youtube-VOS Dataset
A major contribution of this work is the introduction of Refer-Youtube-VOS, the first large-scale dataset for R-VOS. It contains 3,975 videos and nearly 28,000 referring expressions, providing the necessary scale for training robust deep learning models. The dataset includes both "Full-video" descriptions and "First-frame" descriptions to account for different levels of temporal context.
Experimental Results
URVOS achieves state-of-the-art performance across several benchmarks. On the Refer-DAVIS17 dataset, it reached a J-score of 47.29, a significant improvement over previous methods. Ablation studies confirmed the necessity of the Memory Attention Module; removing it led to a substantial drop in performance, highlighting the importance of temporal context in video-level grounding.
Qualitative results show that URVOS is robust to challenging scenarios, including fast motion and complex backgrounds. The researchers also found that performance scales with dataset size, emphasizing the value of the Refer-Youtube-VOS benchmark.
Conclusion and Future Work
URVOS represents a significant step toward intuitive human-AI interaction in video analysis. By unifying language grounding and temporal tracking, it provides a robust framework for referring video object segmentation. Future research will focus on reducing the computational overhead of the memory mechanism and improving the model's ability to handle highly complex spatial relationships in language queries.
Summary updated