Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model

Ruiping Liu, Junwei Zheng, Yufan Chen, Zirui Wang, Kunyu Peng, Kailun Yang, Jiaming Zhang, Marc Pollefeys, Rainer Stiefelhagen

Abstract

Physical environments and circumstances are fundamentally dynamic, yet current 3D datasets and evaluation benchmarks tend to concentrate on either dynamic scenarios or dynamic situations in isolation, resulting in incomplete comprehension. To overcome these constraints, we introduce Situat3DChange, an extensive dataset supporting three situation-aware change understanding tasks following the perception-action model: 121K question-answer pairs, 36K change descriptions for perception tasks, and 17K rearrangement instructions for the action task. To construct this large-scale dataset, Situat3DChange leverages 11K human observations of environmental changes to establish shared mental models and shared situational awareness for human-AI collaboration. These observations, enriched with egocentric and allocentric perspectives as well as categorical and coordinate spatial relations, are integrated using an LLM to support understanding of situated changes. To address the challenge of comparing pairs of point clouds from the same scene with minor changes, we propose SCReasoner, an efficient 3D MLLM approach that enables effective point cloud comparison with minimal parameter overhead and no additional tokens required for the language decoder. Comprehensive evaluation on Situat3DChange tasks highlights both the progress and limitations of MLLMs in dynamic scene and situation understanding.

Introduction

The physical world is never truly static, yet artificial intelligence models often treat 3D environments as frozen snapshots. Current 3D datasets typically focus on purely geometric changes—such as an object's coordinate shift—without considering the human context or the "situation." For a robot or an assistive device, a moved object is not just a change in geometry; it represents a change in a shared mental map. This research introduces the concept of situated change understanding, bridging the gap between perception (identifying what changed) and action (determining how to respond or revert the change).

The Situat3DChange Dataset

Built upon the real-world 3RScan dataset, Situat3DChange contains 903 real-world scan pairs. It leverages 11,000 human observations to generate a massive corpus of situated data. The dataset is designed to support three primary tasks:

  • Perception (QA): 121K Question-Answer pairs covering distance, direction, existence, and counting.
  • Perception (Descriptions): 36K Change Descriptions providing long-form perception from both egocentric (human-centric) and allocentric (object-to-object) perspectives.
  • Action (Rearrangement): 17K Rearrangement Instructions detailing the steps required to revert environmental changes.
Overview of the Situat3DChange dataset, illustrating the comparison between past and present states alongside egocentric views and descriptive QA.

Methods: SCReasoner and the Perception-Action Framework

To process these changes efficiently, the authors propose SCReasoner, a 3D Multimodal Large Language Model (MLLM). Comparing two complex 3D point clouds is computationally expensive and often redundant. SCReasoner addresses this through a selective comparison projector using:

  • Mamba Blocks: These blocks selectively process tokens, identifying only the "deltas" or actual changes between the "past" and "present" scans.
  • Star Operations: Parameter-free element-wise multiplications fuse tokens from different scenes to highlight differences without increasing the token count for the language decoder.

The framework follows a perception-action model, where the AI must first perceive the change and then generate actionable instructions to address it, such as moving an object back to its original location.

A four-step rearrangement instruction sequence demonstrating the action-oriented component of the Situat3DChange dataset.

Key Results and Insights

Evaluation of SCReasoner on Situat3DChange reveals several key findings:

  • State-of-the-Art Performance: SCReasoner outperformed the baseline 3D model (LEO) by 1.2% on change descriptions and 0.6% on rearrangement instructions.
  • 3D vs. 2D Reasoning: Compared to top-tier 2D models like InternVL2, the 3D-native SCReasoner showed massive gains, including a +15.4% improvement on rearrangement tasks. While 2D models are proficient at egocentric distance estimation, they struggle with the allocentric 3D relationships where point-cloud models excel.
  • Human-Centric Alignment: The study highlights a disconnect between "robotic cognition" (Cartesian coordinates) and "human cognition" (relative directions). Models trained on Situat3DChange better align with human-centric spatial reasoning, such as understanding "front" and "back" based on a person's orientation.

Conclusion

Situat3DChange provides a critical benchmark for moving AI beyond static 3D perception toward dynamic, situated understanding. By focusing on informative "deltas" rather than entire scenes, SCReasoner demonstrates that efficient 3D change understanding is possible with minimal computational overhead. Future work will focus on increasing annotation diversity and making the data generation pipeline more domain-agnostic.

Summary updated

Abstract

Physical environments and circumstances are fundamentally dynamic, yet current 3D datasets and evaluation benchmarks tend to concentrate on either dynamic scenarios or dynamic situations in isolation, resulting in incomplete comprehension. To overcome these constraints, we introduce Situat3DChange, an extensive dataset supporting three situation-aware change understanding tasks following the perception-action model: 121K question-answer pairs, 36K change descriptions for perception tasks, and 17K rearrangement instructions for the action task. To construct this large-scale dataset, Situat3DChange leverages 11K human observations of environmental changes to establish shared mental models and shared situational awareness for human-AI collaboration. These observations, enriched with egocentric and allocentric perspectives as well as categorical and coordinate spatial relations, are integrated using an LLM to support understanding of situated changes. To address the challenge of comparing pairs of point clouds from the same scene with minor changes, we propose SCReasoner, an efficient 3D MLLM approach that enables effective point cloud comparison with minimal parameter overhead and no additional tokens required for the language decoder. Comprehensive evaluation on Situat3DChange tasks highlights both the progress and limitations of MLLMs in dynamic scene and situation understanding. Additional experiments on data scaling and cross-domain transfer demonstrate the task-agnostic effectiveness of using Situat3DChange as a training dataset for MLLMs.
BibTeX
@inproceedings{liu2025situat,
  author = {Ruiping Liu and Junwei Zheng and Yufan Chen and Zirui Wang and Kunyu Peng and Kailun Yang and Jiaming Zhang and Marc Pollefeys and Rainer Stiefelhagen},
  title = {{Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=WHARtAg5R9},
}
Figure