M^3-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation

Zixuan Chen, Jiaxin Li, Junxuan Liang, Liming Tan, Yejie Guo, Cewu Lu, Yong-Lu Li

Abstract

Traditional Video Object Segmentation (VOS) models often rely on the assumption that objects maintain a relatively consistent appearance and shape over time. This "rigid body" assumption fails in real-world scenarios where objects undergo complex phase transitions, such as ice melting or glass shattering. We present M³-VOS, a new benchmark designed to evaluate VOS in the presence of these dynamics. Alongside the dataset, we introduce ReVOS, a framework that employs "reverse refinement" to propagate segmentation masks backward from stable end-states to chaotic transition phases. Our results show that ReVOS significantly outperforms existing state-of-the-art models, particularly in challenging cross-phase transitions.

Introduction

Most current VOS benchmarks, such as DAVIS or YouTube-VOS, focus on objects with stable boundaries and morphologies. However, for embodied AI applications like robotic cooking or industrial automation, understanding objects that change state is crucial. When coffee beans are ground or milk is poured, visual characteristics and boundaries change drastically. M³-VOS addresses this gap by providing a diverse set of videos featuring multi-phase, multi-transition, and multi-scenery dynamics.

Defining Phase in Computer Vision

To categorize these complex dynamics, we adopt a macroscopic approach based on visual features rather than strict chemical definitions. Objects are classified into three primary phases:

  • Solid: Further divided into Particulate (capable of splitting), Rigid, and Flexible.
  • Liquid: Categorized as Viscous (stretchy) or Non-Viscous (flowing).
  • Aerosol/Gas: Defined by continuous diffusion and lack of clear boundaries.
Visual taxonomy of phases and transitions used in M³-VOS, illustrating the macroscopic classification of solids, liquids, and gases.

The ReVOS Framework

Phase transitions are inherently entropy-increasing processes, where an ordered state (e.g., a solid block of ice) becomes disordered (e.g., a puddle of water). We hypothesize that reversing the temporal flow can simplify the segmentation task by starting from a more "ordered" or stable state.

ReVOS is a plug-and-play model that implements this "reverse entropy-reduction" through three key components:

  1. Reverse Propagation: Propagates masks backward through a sliding window from stable end-frames.
  2. Booster Module: Enhances mask signals during propagation to maintain tracking of small or diffusing object parts.
  3. Readout Fusion: Merges features from both forward and backward passes to produce a final, refined segmentation.

The M³-VOS Dataset

The M³-VOS benchmark consists of 471 high-resolution videos spanning 14 diverse scenarios, including kitchens, factories, and farms. It covers over 120 object categories and 23 specific phase transitions (13 intra-phase and 10 cross-phase).

Overview of the M³-VOS dataset construction pipeline and the distribution of phase transitions, showing the complexity and variety of state changes captured.

The dataset features dense pixel-level masks at 30 fps. To ensure high-quality annotations, we employed a "dual-model cross-validation" approach, using different AI assistants (SAM2 and Cutie) to assist human annotators and minimize model-specific biases.

Experimental Results

ReVOS achieved a 75.6 J score on the M³-VOS full set, outperforming the previous state-of-the-art Cutie-base (74.6) and significantly exceeding SAM2 (69.5). The performance gains were most pronounced in "Cross-Phase" (CP) transitions, where bidirectional propagation proved essential for handling drastic state changes. Furthermore, ReVOS maintained competitive performance on standard benchmarks like DAVIS '17 (86.0 G score), demonstrating its generalizability to non-transitioning objects.

Conclusion and Future Work

While ReVOS represents a significant step forward, segmenting objects through phase transitions remains an open challenge. Future research should explore integrating physical knowledge—such as fluid dynamics or material properties—into Multimodal Large Language Models (MLLMs) to provide a more robust, "common sense" understanding of object physics in computer vision.

Summary updated

Abstract

Intelligent robots need to interact with diverse objects across various environments. The appearance and state of objects frequently undergo complex transformations depending on the object properties, e.g., phase transitions. However, in the vision community, segmenting dynamic objects with phase transitions is overlooked. In light of this, we introduce the concept of phase in segmentation, which categorizes real-world objects based on their visual characteristics and potential morphological and appearance changes. Then, we present a new benchmark, Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation (M$^3$-VOS), to verify the ability of models to understand object phases, which consists of 479 high-resolution videos spanning over 10 distinct everyday scenarios. It provides dense instance mask annotations that capture both object phases and their transitions. We evaluate state-of-the-art methods on M$^3$-VOS, yielding several key insights. Notably, current appearance-based approaches show significant room for improvement when handling objects with phase transitions. The inherent changes in disorder suggest that the predictive performance of the forward entropy-increasing process can be improved through a reverse entropy-reducing process. These findings lead us to propose ReVOS, a new plug-andplay model that improves its performance by reversal refinement. Our data and code will be publicly available at https://zixuan-chen.github.io/M-cube-VOS.github.io/.
BibTeX
@inproceedings{chen2025m,
  author = {Zixuan Chen and Jiaxin Li and Junxuan Liang and Liming Tan and Yejie Guo and Cewu Lu and Yong-Lu Li},
  title = {{M^3-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02718},
}
Figure