MovSAM: A Single-image Moving Object Segmentation Framework Based on Deep Thinking

Chang Nie, Yiqing Xu, Guangming Wang, Zhe Liu, Yanzi Miao, Hesheng Wang

Abstract

Moving Object Segmentation (MOS) has traditionally been treated as a video-based task, relying on temporal cues and optical flow across multiple frames. However, this dependency introduces vulnerabilities to frame drops, occlusions, and sensor failures. This paper presents MovSAM, a novel framework that achieves state-of-the-art MOS performance using only a single image. By integrating Multimodal Large Language Models (MLLMs) with a "Deep Thinking" iterative reasoning loop, MovSAM mimics human cognitive processes to identify and segment dynamic objects based on semantic context and physical clues. Experimental results demonstrate that MovSAM outperforms several established multi-frame methods on benchmarks such as DAVIS2016 and FBMS.

1. Introduction

In the field of computer vision, identifying moving objects is critical for autonomous navigation and scene understanding. Standard approaches to Moving Object Segmentation (MOS) typically require a sequence of frames to calculate motion vectors. While effective in stable environments, these methods fall into a "temporal trap" when temporal continuity is broken. Furthermore, traditional models often lack the "common sense" required to predict motion intention—such as recognizing that a sprinter in a starting block is a dynamic entity despite being momentarily stationary.

To address these challenges, we introduce MovSAM. Our framework shifts the paradigm from tracking pixel changes to reasoning about motion from static visual evidence. By leveraging the semantic depth of MLLMs, MovSAM identifies motion through non-rigid shapes, suspended postures, and environmental context, providing a robust alternative to flow-based segmentation.

2. Methodology

MovSAM is composed of three integrated modules designed to bridge the gap between static image analysis and dynamic object understanding.

2.1 Searching with Chain-of-Thought (CoT)

The process begins with a Multimodal Large Language Model (specifically Llama 3.2 Vision) analyzing the input image. Using Chain-of-Thought (CoT) prompting, the model searches for visual indicators of motion, such as flowing hair, motion blur, or gravity-defying positions. This reasoning phase culminates in a descriptive text prompt that identifies the target object (e.g., "Segment the runner on the right").

2.2 The Segmentation Module

The segmentation task is handled by a hybrid architecture that combines the high-fidelity boundary precision of SAM2 with the semantic awareness of BEiT-3 (a Vision-Language Model). These features are synthesized through a custom Feature Aggregation Network. This network ensures that the pixel-level segmentation is strictly aligned with the semantic intent of the text prompt generated in the previous step.

2.3 The Deep Thinking Loop

The core innovation of MovSAM is the "Deep Thinking" loop. After an initial segmentation mask is generated, the MLLM inspects the result for logical consistency and completeness. If the mask is fragmented or fails to align with the motion rationale, the model refines its prompt and restarts the segmentation process. The loop terminates only when the model can provide a coherent logical justification for the segmented object's dynamic status, significantly reducing the likelihood of hallucinations.

3. Experimental Evaluation

3.1 Datasets and Setup

MovSAM was evaluated on several standard MOS benchmarks, filtered to assess single-image performance:

  • DAVIS2016: A primary benchmark for video object segmentation.
  • FBMS (Freiburg-Berkeley Motion Segmentation): Used to evaluate the model's generalization capabilities.
  • YouTube-Objects: A diverse dataset covering various categories such as animals and vehicles.
  • Real-World Driving Data: Proprietary data collected via an experimental vehicle to test "pedestrian gaming" scenarios and sensor failure resilience.

3.2 Quantitative Results

MovSAM achieves state-of-the-art (SOTA) results, frequently surpassing methods that utilize temporal data. On the DAVIS2016 benchmark, MovSAM reached a 92.5% J&F score, notably higher than multi-frame competitors like TMO (86.1%) and FlowP-SAM (86.7%). On the FBMS dataset, the model achieved 83.9% J, demonstrating its robustness across varied environments.

3.3 Efficiency and Qualitative Performance

Despite the iterative nature of the Deep Thinking loop, MovSAM maintains an inference time of approximately 0.3 seconds. Qualitatively, the model excels in scenarios involving heavy occlusion and motion blur—areas where traditional optical flow methods typically struggle due to pixel ambiguity.

4. Discussion and Limitations

While MovSAM represents a significant advancement in single-image MOS, several challenges remain. The reliance on large-scale models like Llama 3.2 and BEiT-3 necessitates substantial computational resources, which may limit deployment on edge devices such as drones. Furthermore, while 0.3s is efficient for many applications, high-speed autonomous driving may require further latency reductions. Future work will explore model distillation for real-time edge performance and the extension of this "deep thinking" logic to 3D modalities like LiDAR point clouds.

5. Conclusion

MovSAM demonstrates that semantic reasoning can effectively substitute for temporal cues in moving object segmentation. By teaching AI to "think" before it segments, we move closer to autonomous systems that do not just see motion, but understand the underlying logic of a dynamic scene.

Summary updated

Abstract

Moving object segmentation plays a vital role in understanding dynamic visual environments. While existing methods rely on multi-frame image sequences to identify moving objects, single-image MOS is critical for applications like motion intention prediction and handling camera frame drops. However, segmenting moving objects from a single image remains challenging for existing methods due to the absence of temporal cues. To address this gap, we propose MovSAM, the first framework for single-image moving object segmentation. MovSAM leverages a Multimodal Large Language Model (MLLM) enhanced with Chain-of-Thought (CoT) prompting to search the moving object and generate text prompts based on deep thinking for segmentation. These prompts are cross-fused with visual features from the Segment Anything Model (SAM) and a Vision-Language Model (VLM), enabling logic-driven moving object segmentation. The segmentation results then undergo a deep thinking refinement loop, allowing MovSAM to iteratively improve its understanding of the scene context and inter-object relationships with logical reasoning. This innovative approach enables MovSAM to segment moving objects in single images by considering scene understanding. We implement MovSAM in the real world to validate its practical application and effectiveness for autonomous driving scenarios where the multi-frame methods fail. Furthermore, despite the inherent advantage of multi-frame methods in utilizing temporal information, MovSAM achieves state-of-the-art performance across public MOS benchmarks, reaching 92.5\% on J\&F. Our implementation will be available at https://github.com/IRMVLab/MovSAM.
BibTeX
@inproceedings{nie2025movsam,
  author = {Chang Nie and Yiqing Xu and Guangming Wang and Zhe Liu and Yanzi Miao and Hesheng Wang},
  title = {{MovSAM: A Single-image Moving Object Segmentation Framework Based on Deep Thinking}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
}
Figure