Robust Referring Video Object Segmentation with Cyclic Structural Consensus

Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj, Yan Lu

Abstract

Referring Video Object Segmentation (R-VOS) identifies objects in video sequences based on natural language queries. However, standard R-VOS models often fail when the queried object is absent, leading to "hallucinated" segmentations. This paper introduces $R^2$-VOS (Robust Referring Video Object Segmentation), a task requiring models to handle missing objects. We propose the Cyclic Structural Consensus (CSC) framework, which enforces structural consistency between original and reconstructed text embeddings. Our approach significantly reduces false alarms and achieves state-of-the-art performance on standard and robust benchmarks.

1. Introduction: Addressing the R-VOS Blind Spot

Traditional R-VOS research operates under the assumption that the target object always exists within the video. In real-world scenarios, this assumption is frequently violated. When presented with a "negative" query—where the described object is missing—current models tend to force a segmentation onto irrelevant background elements or other objects, a phenomenon known as hallucination.

Comparison of standard R-VOS and the proposed robust $R^2$-VOS framework on positive and negative samples.

Furthermore, linguistic diversity poses a challenge. Different users may describe the same object using varied phrasing. Existing methods that attempt to align vision and language often use rigid point-to-point matching, which fails to capture the underlying semantic structure of diverse descriptions.

2. Methodology: Cyclic Structural Consensus

To address these issues, we propose the $R^2$-VOS network. The architecture is built around the principle of Semantic Consensus, ensuring that the visual segmentation and the linguistic query are truly aligned.

2.1 Architecture Overview

The model consists of an Object Localizing Module (OLM) that generates a proxy feature to locate the target while suppressing noise. This is followed by a dual-task setup where the model performs referring segmentation while simultaneously attempting to reconstruct the text embedding from visual features.

The $R^2$-VOS architecture (a, b) and the conceptual shift from point-wise (c) to structural consistency (d).

2.2 Structural Consistency

The core innovation is the Cyclic Structural Consensus (CSC). Instead of requiring the reconstructed text embedding to be an exact point-wise match of the original, we enforce structural consistency. As shown in the figure above, we preserve the geometric relationships (distances and angles) between different text expressions in the embedding space. This allows the model to accommodate linguistic flexibility while maintaining a coherent mapping between the visual and textual domains.

2.3 Consensus Discrimination

A dedicated classification head evaluates the "consensus" between the video and text. If the confidence score is low (indicating a negative pair), the model suppresses the segmentation output, effectively preventing false alarms.

3. Experimental Evaluation

We evaluated our model on standard benchmarks (Ref-DAVIS17, Ref-Youtube-VOS) and a newly created $R^2$-Youtube-VOS dataset designed to test robustness against negative queries.

  • Performance: Our model achieved 61.3% $\mathcal{J}\&\mathcal{F}$ on Ref-Youtube-VOS, setting a new state-of-the-art.
  • Robustness: On the $R^2$-Youtube-VOS dataset, our framework demonstrated a significant improvement in the Robustness Score ($\mathcal{R}$), successfully filtering out irrelevant queries where baseline models failed.
  • Efficiency: The system maintains high inference speeds, reaching 30 FPS on a single GPU, making it suitable for real-time applications.

4. Conclusion

By moving beyond the assumption of object presence and adopting structural rather than point-wise consistency, $R^2$-VOS provides a more reliable solution for real-world video understanding. Future work will explore extending this structural consensus to other multi-modal tasks like Video Question Answering where "none of the above" is a necessary response.

Summary updated

Abstract

Referring Video Object Segmentation (R-VOS) is a challenging task that aims to segment an object in a video based on a linguistic expression. Most existing R-VOS methods have a critical assumption: the object referred to must appear in the video. This assumption, which we refer to as "semantic consensus", is often violated in real-world scenarios, where the expression may be queried against false videos. In this work, we highlight the need for a robust R-VOS model that can handle semantic mismatches. Accordingly, we propose an extended task called Robust R-VOS (R 2 -VOS), which accepts unpaired video-text inputs. We tackle this problem by jointly modeling the primary R-VOS problem and its dual (text reconstruction). A structural text-to-text cycle constraint is introduced to discriminate semantic consensus between video-text pairs and impose it in positive pairs, thereby achieving multi-modal alignment from both positive and negative pairs. Our structural constraint effectively addresses the challenge posed by linguistic diversity, overcoming the limitations of previous methods that relied on the point-wise constraint. A new evaluation dataset, R 2 -Youtube-VOS is constructed to measure the model robustness. Our model achieves state-of-the-art performance on R-VOS benchmarks, Ref-DAVIS17 and Ref-Youtube-VOS, and also our R 2 -Youtube-VOS dataset.
BibTeX
@inproceedings{li2023robust,
  author = {Xiang Li and Jinglu Wang and Xiaohao Xu and Xiao Li and Bhiksha Raj and Yan Lu},
  title = {{Robust Referring Video Object Segmentation with Cyclic Structural Consensus}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.02032},
}
Figure