3D Change Localization and Captioning from Dynamic Scans of Indoor Scenes

Yue Qiu, Shintaro Yamamoto, Ryosuke Yamada, Ryota Suzuki, Hirokatsu Kataoka, Kenji Iwata, Yutaka Satoh

Introduction

Understanding how environments evolve over time is a fundamental capability for autonomous agents. While traditional computer vision has explored "change captioning" using 2D image pairs, these methods often fail to capture the full spatial context of 3D environments. This paper introduces a novel task: 3D Change Localization and Captioning. Unlike 2D approaches, this task requires identifying the precise 3D coordinates of a change and generating a natural language description (e.g., "The fridge on the corner of the room has been opened").

Overview of the 3D change localization and captioning pipeline, showing observation routes, point clouds, and final model output.

The DenseChangeCap Framework

To address this challenge, the authors propose DenseChangeCap, an end-to-end framework designed to process dynamic 3D scans. The pipeline begins with Point Cloud Differencing, comparing "before" and "after" registered scans to isolate regions of interest.

Geometric features are extracted using a PointNet++ backbone. The framework is inherently modular, allowing for different 3D object detectors—such as the transformer-based 3DETR or the voting-based VoteNet—to propose bounding boxes for detected changes. Finally, a captioning head (LSTM or Transformer) generates descriptions based on five atomic change types: add, delete, move, open, and close.

Datasets: DyS2Change and SUNRGBD2Change

A major contribution of this work is the release of DyS2Change, a massive synthetic dataset generated via the AI2THOR simulator. It comprises 120 indoor scenes with over 37,000 change pairs and 661,000 captions. This dataset provides the scale necessary for deep learning models to learn complex spatial relationships. To evaluate real-world applicability, the authors also introduced SUNRGBD2Change, a dataset derived from real-world SUN RGB-D scans, facilitating sim-to-real transfer studies.

Results and Discussion

The experiments demonstrate that 3D context significantly enhances change understanding. A key finding is the effectiveness of simulation-based pretraining: models pretrained on DyS2Change showed a +12.8% improvement in mAP0.25 when tested on real-world data compared to those trained from scratch.

Close-up of model output showing localized changes (fridge and microwave) with corresponding natural language captions.

In terms of performance, the 3DETR-based model achieved a 35.3 mAP0.25 on the synthetic test set. Captioning quality reached a CIDER score of 57.3, establishing a robust baseline for future research. While the model performs well on large objects, it still faces challenges with sparse point clouds of smaller items and occasionally confuses similar change states (e.g., "moved" vs. "missing").

Conclusion

This research bridges the gap between 3D vision and natural language processing. By providing both a modular framework and a large-scale dataset, the authors pave the way for AI systems that can more effectively navigate and understand the dynamic nature of human-centric indoor spaces. Future work will focus on improving sensitivity to small objects and handling higher frequencies of simultaneous changes.

Summary updated

Abstract

Daily indoor scenes often involve constant changes due to human activities. To recognize scene changes, existing change captioning methods focus on describing changes from two images of a scene. However, to accurately perceive and appropriately evaluate physical changes and then identify the geometry of changed objects, recognizing and localizing changes in 3D space is crucial. Therefore, we propose a task to explicitly localize changes in 3D bounding boxes from two point clouds and describe detailed scene changes, including change types, object attributes, and spatial locations. Moreover, we create a simulated dataset with various scenes, allowing generating data without labor costs. We further propose a framework that allows different 3D object detectors to be incorporated in the change detection process, after which captions are generated based on the correlations of different change regions. The proposed framework achieves promising results in both change detection and captioning. Furthermore, we also evaluated on data collected from real scenes. The experiments show that pretraining on the proposed dataset increases the change detection accuracy by +12.8% (mAP0.25) when applied to real-world data. We believe that our proposed dataset and discussion could provide both a new benchmark and in-sights for future studies in scene change understanding.
BibTeX
@inproceedings{qiu2023d,
  author = {Yue Qiu and Shintaro Yamamoto and Ryosuke Yamada and Ryota Suzuki and Hirokatsu Kataoka and Kenji Iwata and Yutaka Satoh},
  title = {{3D Change Localization and Captioning from Dynamic Scans of Indoor Scenes}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00123},
}
Figure