Abstract
Referring Video Object Segmentation (R-VOS) identifies objects in video sequences based on natural language queries. However, standard R-VOS models often fail when the queried object is absent, leading to "hallucinated" segmentations. This paper introduces $R^2$-VOS (Robust Referring Video Object Segmentation), a task requiring models to handle missing objects. We propose the Cyclic Structural Consensus (CSC) framework, which enforces structural consistency between original and reconstructed text embeddings. Our approach significantly reduces false alarms and achieves state-of-the-art performance on standard and robust benchmarks.
1. Introduction: Addressing the R-VOS Blind Spot
Traditional R-VOS research operates under the assumption that the target object always exists within the video. In real-world scenarios, this assumption is frequently violated. When presented with a "negative" query—where the described object is missing—current models tend to force a segmentation onto irrelevant background elements or other objects, a phenomenon known as hallucination.
Furthermore, linguistic diversity poses a challenge. Different users may describe the same object using varied phrasing. Existing methods that attempt to align vision and language often use rigid point-to-point matching, which fails to capture the underlying semantic structure of diverse descriptions.
2. Methodology: Cyclic Structural Consensus
To address these issues, we propose the $R^2$-VOS network. The architecture is built around the principle of Semantic Consensus, ensuring that the visual segmentation and the linguistic query are truly aligned.
2.1 Architecture Overview
The model consists of an Object Localizing Module (OLM) that generates a proxy feature to locate the target while suppressing noise. This is followed by a dual-task setup where the model performs referring segmentation while simultaneously attempting to reconstruct the text embedding from visual features.
2.2 Structural Consistency
The core innovation is the Cyclic Structural Consensus (CSC). Instead of requiring the reconstructed text embedding to be an exact point-wise match of the original, we enforce structural consistency. As shown in the figure above, we preserve the geometric relationships (distances and angles) between different text expressions in the embedding space. This allows the model to accommodate linguistic flexibility while maintaining a coherent mapping between the visual and textual domains.
2.3 Consensus Discrimination
A dedicated classification head evaluates the "consensus" between the video and text. If the confidence score is low (indicating a negative pair), the model suppresses the segmentation output, effectively preventing false alarms.
3. Experimental Evaluation
We evaluated our model on standard benchmarks (Ref-DAVIS17, Ref-Youtube-VOS) and a newly created $R^2$-Youtube-VOS dataset designed to test robustness against negative queries.
- Performance: Our model achieved 61.3% $\mathcal{J}\&\mathcal{F}$ on Ref-Youtube-VOS, setting a new state-of-the-art.
- Robustness: On the $R^2$-Youtube-VOS dataset, our framework demonstrated a significant improvement in the Robustness Score ($\mathcal{R}$), successfully filtering out irrelevant queries where baseline models failed.
- Efficiency: The system maintains high inference speeds, reaching 30 FPS on a single GPU, making it suitable for real-time applications.
4. Conclusion
By moving beyond the assumption of object presence and adopting structural rather than point-wise consistency, $R^2$-VOS provides a more reliable solution for real-world video understanding. Future work will explore extending this structural consensus to other multi-modal tasks like Video Question Answering where "none of the above" is a necessary response.
Summary updated