Abstract
Moving Object Segmentation (MOS) has traditionally been treated as a video-based task, relying on temporal cues and optical flow across multiple frames. However, this dependency introduces vulnerabilities to frame drops, occlusions, and sensor failures. This paper presents MovSAM, a novel framework that achieves state-of-the-art MOS performance using only a single image. By integrating Multimodal Large Language Models (MLLMs) with a "Deep Thinking" iterative reasoning loop, MovSAM mimics human cognitive processes to identify and segment dynamic objects based on semantic context and physical clues. Experimental results demonstrate that MovSAM outperforms several established multi-frame methods on benchmarks such as DAVIS2016 and FBMS.
1. Introduction
In the field of computer vision, identifying moving objects is critical for autonomous navigation and scene understanding. Standard approaches to Moving Object Segmentation (MOS) typically require a sequence of frames to calculate motion vectors. While effective in stable environments, these methods fall into a "temporal trap" when temporal continuity is broken. Furthermore, traditional models often lack the "common sense" required to predict motion intention—such as recognizing that a sprinter in a starting block is a dynamic entity despite being momentarily stationary.
To address these challenges, we introduce MovSAM. Our framework shifts the paradigm from tracking pixel changes to reasoning about motion from static visual evidence. By leveraging the semantic depth of MLLMs, MovSAM identifies motion through non-rigid shapes, suspended postures, and environmental context, providing a robust alternative to flow-based segmentation.
2. Methodology
MovSAM is composed of three integrated modules designed to bridge the gap between static image analysis and dynamic object understanding.
2.1 Searching with Chain-of-Thought (CoT)
The process begins with a Multimodal Large Language Model (specifically Llama 3.2 Vision) analyzing the input image. Using Chain-of-Thought (CoT) prompting, the model searches for visual indicators of motion, such as flowing hair, motion blur, or gravity-defying positions. This reasoning phase culminates in a descriptive text prompt that identifies the target object (e.g., "Segment the runner on the right").
2.2 The Segmentation Module
The segmentation task is handled by a hybrid architecture that combines the high-fidelity boundary precision of SAM2 with the semantic awareness of BEiT-3 (a Vision-Language Model). These features are synthesized through a custom Feature Aggregation Network. This network ensures that the pixel-level segmentation is strictly aligned with the semantic intent of the text prompt generated in the previous step.
2.3 The Deep Thinking Loop
The core innovation of MovSAM is the "Deep Thinking" loop. After an initial segmentation mask is generated, the MLLM inspects the result for logical consistency and completeness. If the mask is fragmented or fails to align with the motion rationale, the model refines its prompt and restarts the segmentation process. The loop terminates only when the model can provide a coherent logical justification for the segmented object's dynamic status, significantly reducing the likelihood of hallucinations.
3. Experimental Evaluation
3.1 Datasets and Setup
MovSAM was evaluated on several standard MOS benchmarks, filtered to assess single-image performance:
- DAVIS2016: A primary benchmark for video object segmentation.
- FBMS (Freiburg-Berkeley Motion Segmentation): Used to evaluate the model's generalization capabilities.
- YouTube-Objects: A diverse dataset covering various categories such as animals and vehicles.
- Real-World Driving Data: Proprietary data collected via an experimental vehicle to test "pedestrian gaming" scenarios and sensor failure resilience.
3.2 Quantitative Results
MovSAM achieves state-of-the-art (SOTA) results, frequently surpassing methods that utilize temporal data. On the DAVIS2016 benchmark, MovSAM reached a 92.5% J&F score, notably higher than multi-frame competitors like TMO (86.1%) and FlowP-SAM (86.7%). On the FBMS dataset, the model achieved 83.9% J, demonstrating its robustness across varied environments.
3.3 Efficiency and Qualitative Performance
Despite the iterative nature of the Deep Thinking loop, MovSAM maintains an inference time of approximately 0.3 seconds. Qualitatively, the model excels in scenarios involving heavy occlusion and motion blur—areas where traditional optical flow methods typically struggle due to pixel ambiguity.
4. Discussion and Limitations
While MovSAM represents a significant advancement in single-image MOS, several challenges remain. The reliance on large-scale models like Llama 3.2 and BEiT-3 necessitates substantial computational resources, which may limit deployment on edge devices such as drones. Furthermore, while 0.3s is efficient for many applications, high-speed autonomous driving may require further latency reductions. Future work will explore model distillation for real-time edge performance and the extension of this "deep thinking" logic to 3D modalities like LiDAR point clouds.
5. Conclusion
MovSAM demonstrates that semantic reasoning can effectively substitute for temporal cues in moving object segmentation. By teaching AI to "think" before it segments, we move closer to autonomous systems that do not just see motion, but understand the underlying logic of a dynamic scene.
Summary updated