Introduction
Understanding how environments evolve over time is a fundamental capability for autonomous agents. While traditional computer vision has explored "change captioning" using 2D image pairs, these methods often fail to capture the full spatial context of 3D environments. This paper introduces a novel task: 3D Change Localization and Captioning. Unlike 2D approaches, this task requires identifying the precise 3D coordinates of a change and generating a natural language description (e.g., "The fridge on the corner of the room has been opened").
The DenseChangeCap Framework
To address this challenge, the authors propose DenseChangeCap, an end-to-end framework designed to process dynamic 3D scans. The pipeline begins with Point Cloud Differencing, comparing "before" and "after" registered scans to isolate regions of interest.
Geometric features are extracted using a PointNet++ backbone. The framework is inherently modular, allowing for different 3D object detectors—such as the transformer-based 3DETR or the voting-based VoteNet—to propose bounding boxes for detected changes. Finally, a captioning head (LSTM or Transformer) generates descriptions based on five atomic change types: add, delete, move, open, and close.
Datasets: DyS2Change and SUNRGBD2Change
A major contribution of this work is the release of DyS2Change, a massive synthetic dataset generated via the AI2THOR simulator. It comprises 120 indoor scenes with over 37,000 change pairs and 661,000 captions. This dataset provides the scale necessary for deep learning models to learn complex spatial relationships. To evaluate real-world applicability, the authors also introduced SUNRGBD2Change, a dataset derived from real-world SUN RGB-D scans, facilitating sim-to-real transfer studies.
Results and Discussion
The experiments demonstrate that 3D context significantly enhances change understanding. A key finding is the effectiveness of simulation-based pretraining: models pretrained on DyS2Change showed a +12.8% improvement in mAP0.25 when tested on real-world data compared to those trained from scratch.
In terms of performance, the 3DETR-based model achieved a 35.3 mAP0.25 on the synthetic test set. Captioning quality reached a CIDER score of 57.3, establishing a robust baseline for future research. While the model performs well on large objects, it still faces challenges with sparse point clouds of smaller items and occasionally confuses similar change states (e.g., "moved" vs. "missing").
Conclusion
This research bridges the gap between 3D vision and natural language processing. By providing both a modular framework and a large-scale dataset, the authors pave the way for AI systems that can more effectively navigate and understand the dynamic nature of human-centric indoor spaces. Future work will focus on improving sensitivity to small objects and handling higher frequencies of simultaneous changes.
Summary updated