Abstract
Traditional Video Object Segmentation (VOS) models often rely on the assumption that objects maintain a relatively consistent appearance and shape over time. This "rigid body" assumption fails in real-world scenarios where objects undergo complex phase transitions, such as ice melting or glass shattering. We present M³-VOS, a new benchmark designed to evaluate VOS in the presence of these dynamics. Alongside the dataset, we introduce ReVOS, a framework that employs "reverse refinement" to propagate segmentation masks backward from stable end-states to chaotic transition phases. Our results show that ReVOS significantly outperforms existing state-of-the-art models, particularly in challenging cross-phase transitions.
Introduction
Most current VOS benchmarks, such as DAVIS or YouTube-VOS, focus on objects with stable boundaries and morphologies. However, for embodied AI applications like robotic cooking or industrial automation, understanding objects that change state is crucial. When coffee beans are ground or milk is poured, visual characteristics and boundaries change drastically. M³-VOS addresses this gap by providing a diverse set of videos featuring multi-phase, multi-transition, and multi-scenery dynamics.
Defining Phase in Computer Vision
To categorize these complex dynamics, we adopt a macroscopic approach based on visual features rather than strict chemical definitions. Objects are classified into three primary phases:
- Solid: Further divided into Particulate (capable of splitting), Rigid, and Flexible.
- Liquid: Categorized as Viscous (stretchy) or Non-Viscous (flowing).
- Aerosol/Gas: Defined by continuous diffusion and lack of clear boundaries.
The ReVOS Framework
Phase transitions are inherently entropy-increasing processes, where an ordered state (e.g., a solid block of ice) becomes disordered (e.g., a puddle of water). We hypothesize that reversing the temporal flow can simplify the segmentation task by starting from a more "ordered" or stable state.
ReVOS is a plug-and-play model that implements this "reverse entropy-reduction" through three key components:
- Reverse Propagation: Propagates masks backward through a sliding window from stable end-frames.
- Booster Module: Enhances mask signals during propagation to maintain tracking of small or diffusing object parts.
- Readout Fusion: Merges features from both forward and backward passes to produce a final, refined segmentation.
The M³-VOS Dataset
The M³-VOS benchmark consists of 471 high-resolution videos spanning 14 diverse scenarios, including kitchens, factories, and farms. It covers over 120 object categories and 23 specific phase transitions (13 intra-phase and 10 cross-phase).
The dataset features dense pixel-level masks at 30 fps. To ensure high-quality annotations, we employed a "dual-model cross-validation" approach, using different AI assistants (SAM2 and Cutie) to assist human annotators and minimize model-specific biases.
Experimental Results
ReVOS achieved a 75.6 J score on the M³-VOS full set, outperforming the previous state-of-the-art Cutie-base (74.6) and significantly exceeding SAM2 (69.5). The performance gains were most pronounced in "Cross-Phase" (CP) transitions, where bidirectional propagation proved essential for handling drastic state changes. Furthermore, ReVOS maintained competitive performance on standard benchmarks like DAVIS '17 (86.0 G score), demonstrating its generalizability to non-transitioning objects.
Conclusion and Future Work
While ReVOS represents a significant step forward, segmenting objects through phase transitions remains an open challenge. Future research should explore integrating physical knowledge—such as fluid dynamics or material properties—into Multimodal Large Language Models (MLLMs) to provide a more robust, "common sense" understanding of object physics in computer vision.
Summary updated