M3-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation

Zixuan Chen, Jiaxin Li, Liming Tan, Yejie Guo, Junxuan Liang, Cewu Lu, Yong-Lu Li

Introduction

Video Object Segmentation (VOS) has achieved remarkable success in tracking rigid or semi-rigid objects like vehicles and pedestrians. However, real-world environments are dynamic, involving materials that undergo significant physical transformations. Intelligent systems must navigate scenarios where objects change state—such as ice melting into water, balloons bursting into fragments, or smoke diffusing into the air. Traditional VOS models, which rely heavily on appearance consistency, often fail when an object's morphology and physical phase shift drastically.

To address this gap, the M3-VOS framework introduces a comprehensive benchmark and a novel "reverse-refinement" model, ReVOS, designed to maintain object identity through complex physical transitions.

The M3-VOS Benchmark

The authors identify a critical lack of high-quality data representing physical state changes. To solve this, they curated the M3-VOS dataset, which significantly expands the scope of existing benchmarks like DAVIS or YouTube-VOS.

Dataset Composition

The benchmark consists of 479 high-resolution videos containing over 205,000 dense instance masks. It covers 14 diverse scenarios (e.g., kitchens, laboratories, farms) and more than 120 object categories. Crucially, the dataset labels 23 distinct types of transitions:

  • Intra-phase transitions (13 types): Changes within the same state, such as stretching a flexible material or splitting a solid.
  • Cross-phase transitions (10 types): Changes between states, such as solidification (liquid to solid) or sublimation (solid to gas).
Taxonomy of physical phases and transitions in the M3-VOS benchmark.

Methodology: ReVOS and the Entropy Hypothesis

The core technical innovation is ReVOS (Reverse VOS), a plug-and-play module that leverages bidirectional propagation.

The Entropy Hypothesis

The researchers observed that as an object undergoes a transition (like shattering), its "disorder" or entropy increases over time. Using Local Binary Patterns (LBP) to quantify mask complexity, they proved that tracking a "messy" end-state backward toward a "clean" initial state is often more stable than forward tracking. In the reverse direction, the process effectively "reduces" disorder, providing a clearer signal for mask refinement.

Architecture

ReVOS performs a forward pass to establish initial temporal context and then uses the final state to propagate masks backward. To integrate these passes, the model employs a Booster Module to amplify mask signals and a Readout Fusion block to merge features from both temporal directions.

The M3-VOS data collection and validation pipeline, including scene distribution and dual-model cross-validation.

Experimental Results

ReVOS sets a new state-of-the-art on the M3-VOS benchmark, particularly excelling in the most challenging scenarios.

  • Overall Performance: ReVOS achieved a Jaccard index ($\mathcal{J}$) of 75.6, outperforming top-tier models like Cutie-base (74.6) and SAM2 (69.5).
  • Cross-Phase Mastery: In "Cross-Phase" (CP) transitions, where appearance changes are most extreme, ReVOS reached a $\mathcal{J}$ of 78.1, demonstrating the robustness of the bidirectional approach.
  • Efficiency: The model maintains a frame rate of 15-21 FPS, making it a practical enhancement for existing VOS backbones.

Conclusion and Future Directions

The M3-VOS benchmark and ReVOS model represent a significant step toward "physics-aware" computer vision. While the model handles state changes effectively, challenges remain in multi-instance scenarios where an object splits into many identical parts. Future work may involve integrating Multimodal Large Language Models (MLLMs) to provide common-sense physical priors—such as gravity or fluid dynamics—to further guide the segmentation process in complex, real-world environments.

Summary updated

BibTeX
@inproceedings{chen2025m,
  author = {Zixuan Chen and Jiaxin Li and Liming Tan and Yejie Guo and Junxuan Liang and Cewu Lu and Yong-Lu Li},
  title = {{M3-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02718},
}
Figure