Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model

Ruiping Liu, Junwei Zheng, Yufan Chen, Zirui Wang, Kunyu Peng, Kailun Yang, Jiaming Zhang, Marc Pollefeys, Rainer Stiefelhagen

TL;DR

Existing 3D vision-language benchmarks treat dynamic environments and situated human perspectives in isolation, failing to reflect how humans perceive physical scene changes. Situat3DChange bridges this gap with a large-scale situated 3D change dataset grounded in 11K human annotations, and introduces SCReasoner, a token-efficient 3D multimodal large language model (MLLM) that uses selective state-space modeling (Mamba) and star operations to spot physical transformations between paired point clouds without language-decoder bloat.

What Problem Does This Paper Address?

Physical spaces and observer circumstances constantly change—a chair is moved slightly, creating an obstacle, or items are rearranged. However, current embodied AI and 3D vision research typically separates these dynamics:

  1. Dynamic scenes vs. dynamic situations: Existing datasets focus either on static scenes from a situated viewpoint (e.g., SQA3D) or synthetic change detection lacking human situational context (e.g., Dy2Change).
  2. Cognitive misalignment: Conventional automated pipelines rely on 3D bounding boxes and scene graphs. These lack sensitivity to subtle geometric shifts (such as a 10 cm displacement or slight rotation) and encode space in Cartesian coordinates, whereas humans intuitively represent space egocentrically (e.g., clock directions, cylindrical coordinates, and step counts).
  3. Model token redundancy: Directly feeding two nearly identical 3D point clouds into an MLLM by prepending tokens overwhelms the language decoder with redundant information rather than highlighting changes.

Methods

To address these challenges, the authors design a human-aligned data generation pipeline and a tailored 3D MLLM architecture:

[Scan Pre-Change]  ───┐
                      ├──> Shared Point Cloud Encoder ──> Selective Projector (Mamba) ──┐
[Scan Post-Change] ───┘                                                                ├──> Star Fusion (*) ──> LLM Decoder
                                                      Current Scene Tokens  ───────────┘
  • Perception-Action Data Pipeline: Starting with 11K expert human annotations on real-world indoor scans, the authors capture situation-agnostic reasons, descriptions, warnings, and rearrangement steps. Using geometric situation sampling (sitting, standing, interacting) and an LLM conditioned on an Object-centric Chain-of-Thought (O-CoT), they synthesize situated long-form text and concise QA pairs.
  • SCReasoner Model:
  • Shared 3D Encoder: Both pre- and post-change point clouds share an encoder to extract feature tokens.
  • Selective Projector & Star Fusion: Rather than concatenating all tokens, SCReasoner passes previous-scene tokens through a selective Mamba module to isolate change-relevant features, then fuses them with the current scene tokens using parameter-free star operations (element-wise multiplication). This yields zero extra token overhead for the language decoder.
  • Revised Distance Metric: To evaluate numerical spatial reasoning without division-by-zero artifacts when stationary objects have a ground-truth displacement of zero ($d_{\text{gt}} = 0$), the authors introduce a revised relative error metric bounded between 0 and 1.

Datasets

  • Situat3DChange (Introduced): Built on 903 scan pairs from real-world 3RScan scenes. Contains 174K total instances divided into:
  • 121K concise QA pairs spanning 9 types (egocentric distance/direction, warnings, allocentric displacements/relationships, and object attributes/counting).
  • 36K change descriptions (perception).
  • 17K rearrangement instructions (action).
  • Evaluation & Transfer Baselines: Evaluated against 2D MLLMs (InternVL2, DeepSeek-VL, Qwen2.5-VL, LLaVA-NeXT) and 3D MLLMs (LEO). Transferability is tested on ScanNet benchmarks: Scan2Cap, ScanQA, and SQA3D.

Key Insights and Ideas

  • Cylindrical and Clock-Face Mental Models Matter: Humans describe spatial relationships relative to their facing direction using clock orientations and steps, not Cartesian offsets. Aligning MLLM training data with this structure markedly improves qualitative relevance.
  • Selective Comparison Beats Brute-Force Prepending: When two point clouds share 95%+ identical structure, prepending all tokens dilutes the change signal. A selective sequence model (Mamba) effectively filters out static background noise.
  • 2D vs. 3D Trade-Offs: 2D panoramic MLLMs achieve competitive scores on concise egocentric orientation QA, but fail on long-form change descriptions and allocentric tasks due to occlusions and a lack of true 3D spatial grounding.

Key Results

  • Long-Form Tasks: On change description and rearrangement instruction, SCReasoner achieves GPT-based evaluation scores of 13.9 and 30.7, outperforming the 3D baseline LEO (12.7 / 30.1) and fine-tuned 2D InternVL2-7B (8.2 / 16.3).
  • Concise QA: SCReasoner reaches an average score of 53.8%, excelling particularly in allocentric relationships (63.5%) and allocentric displacement (43.5%), outperforming LEO (50.2% and 40.5%).
  • Cross-Domain Generalization: Fine-tuning on both ScanNet benchmarks and Situat3DChange ($S3C + SN$) improves general 3D grounding over training on ScanNet alone, pushing Scan2Cap CIDEr from 82.2 to 83.4 and ScanQA CIDEr from 103.1 to 104.3.

Limitations and Future Work

  • Annotation Diversity: Each change instance in the raw dataset was annotated by only one human annotator, limiting linguistic variation.
  • Test Set Availability: Because 3RScan lacks public test-set annotations, evaluation relies on a hold-out split from the validation set.
  • Rearrangement Scaling: The rearrangement task showed inconsistent scaling trends with data volume, likely caused by coarser instruction units (steps and generic left/right cues) and a smaller overall sample size.

Summary updated

Abstract

Physical environments and circumstances are fundamentally dynamic, yet current 3D datasets and evaluation benchmarks tend to concentrate on either dynamic scenarios or dynamic situations in isolation, resulting in incomplete comprehension. To overcome these constraints, we introduce Situat3DChange, an extensive dataset supporting three situation-aware change understanding tasks following the perception-action model: 121K question-answer pairs, 36K change descriptions for perception tasks, and 17K rearrangement instructions for the action task. To construct this large-scale dataset, Situat3DChange leverages 11K human observations of environmental changes to establish shared mental models and shared situational awareness for human-AI collaboration. These observations, enriched with egocentric and allocentric perspectives as well as categorical and coordinate spatial relations, are integrated using an LLM to support understanding of situated changes. To address the challenge of comparing pairs of point clouds from the same scene with minor changes, we propose SCReasoner, an efficient 3D MLLM approach that enables effective point cloud comparison with minimal parameter overhead and no additional tokens required for the language decoder. Comprehensive evaluation on Situat3DChange tasks highlights both the progress and limitations of MLLMs in dynamic scene and situation understanding. Additional experiments on data scaling and cross-domain transfer demonstrate the task-agnostic effectiveness of using Situat3DChange as a training dataset for MLLMs. The established dataset and source code are publicly available at: https://github.com/RuipingL/Situat3DChange.
BibTeX
@inproceedings{liu2025situat,
  author = {Ruiping Liu and Junwei Zheng and Yufan Chen and Zirui Wang and Kunyu Peng and Kailun Yang and Jiaming Zhang and Marc Pollefeys and Rainer Stiefelhagen},
  title = {{Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=WHARtAg5R9},
}

Loading related content...

Figure