Abstract
Daily objects commonly experience state changes. For example, slicing a cucumber changes its state from whole to sliced. Learning about object state changes in Video Object Segmentation (VOS) is crucial for understanding and interacting with the visual world. Conventional VOS benchmarks do not consider this challenging yet crucial problem. This paper makes a pioneering effort to introduce a weakly-supervised benchmark on Video State-Changing Object Segmentation (VSCOS). We construct our VSCOS benchmark by selecting state-changing videos from existing datasets. In advocate of an annotation-efficient approach towards state-changing object segmentation, we only annotate the first and last frames of training videos, which is different from conventional VOS. Notably, an open-vocabulary setting is included to evaluate the generalization to novel types of objects or state changes. We empirically illustrate that state-of-the-art VOS models struggle with state-changing objects and lose track after the state changes. We analyze the main difficulties of our VSCOS task and identify three technical improvements, namely, fine-tuning strategies, representation learning, and integrating motion information. Applying these improvements results in a strong baseline for segmenting state-changing objects consistently.
Introduction: The Identity Crisis in VOS
Current Video Object Segmentation (VOS) models are highly effective at tracking objects that maintain a consistent appearance. Standard VOS is essentially a sophisticated matching game: given a mask for an object in the initial frame, the model tracks that same "look" throughout the video. This paradigm works well for objects like a walking animal or a moving car, where the visual signature remains relatively stable.
However, the real world is transformative. When an egg is cracked, milk is poured, or a vegetable is chopped, the object’s visual signature changes fundamentally. State-of-the-art models like XMem often lose track of objects the moment such a state change occurs because they lack a "common sense" understanding of object identity across states. This paper formalizes the Video State-Changing Object Segmentation (VSCOS) task, requiring models to segment objects through rigid decomposition (cutting), non-rigid transformation (pouring), or appearance changes (cooking).
Methodology: Bridging the State Gap
To address the limitations of existing models, the authors propose a model-agnostic baseline that can be integrated into current VOS architectures (e.g., XMem, DeAOT). Their approach is built on three key pillars:
- Cycle-Consistent Fine-Tuning: Recognizing that state changes are logically "reversible" (the slices still constitute the original cucumber), the model is trained to predict the final frame mask from the first, and then—in a reverse pass—predict the first frame mask from the last.
- Teacher-Student Learning: To ensure stability and prevent the model from predicting empty masks for intermediate frames, an Exponential Moving Average (EMA) teacher model guides the student model's middle-frame predictions.
- Representation Learning (CRW): A Contrastive Random Walk (CRW) loss is integrated to help the model learn that features from the "whole" object and the "transformed" object should remain close in embedding space, despite their visual differences.
- Motion Integration: Optical flow features are optionally fused into the encoder to provide cues that fragments moving together likely belong to the same original entity.
The VSCOS Benchmark
The authors introduced the VSCOS Benchmark, primarily derived from the EPIC-Kitchens dataset, to evaluate this new task:
- Scale: 1,905 video clips covering 30 action categories and 124 object categories.
- Weak Supervision: In an annotation-efficient approach, only the first and last frames are masked during training, forcing the model to learn the underlying transformation.
- Open-Vocabulary Setting: The benchmark includes a test set with novel objects and state changes to evaluate generalization to unseen scenarios.
Evaluation and Insights
The core challenge identified is that VOS is not just a tracking problem; it is an identity problem. A model relying solely on pixel matching cannot understand that a pile of diced onions is the same entity as the original bulb.
To accurately measure performance on fragmented objects, the authors propose the connected component Jaccard index ($cc\mathcal{J}$). This metric evaluates how well the model captures every individual fragment, ensuring that small pieces are not ignored in favor of the largest part of the object.
Results
The performance gap between standard models and the proposed baseline is significant. On the VSCOS benchmark, the baseline XMem model's performance drops to 66.7 $\mathcal{J}\&\mathcal{F}$. By applying the proposed "State-Change" (SC) adaptations:
- XMem-SC improved to 76.5 $\mathcal{J}\&\mathcal{F}$.
- DeAOT-SC reached 77.1 $\mathcal{J}\&\mathcal{F}$.
- In the Open-Vocabulary test, XMem-SC achieved 66.2, compared to 57.4 for the baseline, demonstrating a generalizable understanding of transformations.
Conclusion and Future Work
While the proposed adaptations significantly improve performance, challenges remain. Models still struggle with over-segmentation (e.g., including tools like knives in the object mask) and missing parts in cluttered scenes. Future research may explore integrating 3D geometry or causal reasoning to better understand the mechanics of state changes, moving beyond simple observation to a deeper understanding of object transformations.
Summary updated