SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment

Qi Xu, Dongxu Wei, Lingzhe Zhao, Wenpu Li, Zhangchi Huang, Shunping Ji, Peidong Liu

Abstract

SIU3R is a novel "alignment-free" framework that performs high-quality 3D reconstruction and multi-task scene understanding simultaneously in a single feed-forward pass. By bypassing the traditional requirement of aligning 3D features with 2D vision-language models, it achieves state-of-the-art performance in both geometry and semantics while running in approximately 0.13 seconds.

Introduction

Traditionally, 3D reconstruction—the process of building a geometric model—and scene understanding—identifying the objects within that model—have been treated as separate pipelines. Recent attempts at simultaneous processing usually rely on a 2D-to-3D feature alignment paradigm. These methods "lift" features from pre-trained 2D Vision-Language Models (VLMs) like CLIP into 3D representations such as NeRF or 3D Gaussian Splatting.

However, this approach faces three major bottlenecks:

  1. Limited Instance Awareness: 2D VLMs struggle to distinguish individual object instances, hindering instance and panoptic segmentation.
  2. Information Loss: Memory constraints often force heavy compression of 2D features, discarding fine-grained semantic details.
  3. Efficiency: Most existing methods require hours of per-scene optimization, making them impractical for real-time robotics or embodied AI.
Comparison of the traditional feature alignment paradigm (a) with the proposed alignment-free paradigm (b), and an overview of SIU3R applications (c).

The Alignment-Free Paradigm

SIU3R introduces an alignment-free paradigm that bridges reconstruction and understanding through a pixel-aligned 3D representation. Unlike previous methods that match features to a fixed 2D model, SIU3R utilizes a set of learnable Unified Queries. These queries are shared across different views and tasks, including semantic, instance, panoptic, and text-referred segmentation, ensuring consistency without an external VLM anchor.

A core innovation is the Mutual Benefit Mechanism, where reconstruction and understanding reinforce each other:

  • Reconstruction $\rightarrow$ Understanding: Reconstructed 3D geometry (3D Gaussians) guides the aggregation of semantic information across views, resolving inconsistencies from occlusions or lighting.
  • Understanding $\rightarrow$ Reconstruction: Predicted segmentation masks enforce "intra-instance depth continuity," ensuring that the 3D geometry of an object (like a chair) remains structurally continuous.

Key Insights: Bidirectional Synergy

The "Aha" moment of SIU3R is that alignment with a 2D model is not a prerequisite for 3D understanding. By allowing the model to learn its own unified queries, it bypasses the semantic bottlenecks and information loss inherent in feature alignment. Furthermore, SIU3R demonstrates that multi-tasking can be a "win-win": improved geometry leads to more consistent segmentation, while clear object boundaries result in sharper, more accurate 3D reconstructions.

Experimental Results and Applications

SIU3R was evaluated on the ScanNet dataset, with pre-training on COCO to establish a strong baseline for object detection. The results demonstrate significant improvements over previous state-of-the-art methods like LSM:

  • Superior Accuracy: SIU3R achieved a PSNR of 25.96 (vs. 21.88) for reconstruction and an mIoU of 0.5922 (vs. 0.2745) for understanding.
  • Blazing Speed: The model completes both tasks in roughly 0.13 seconds, a massive leap over the minutes or hours required by optimization-based methods.
SIU3R enables multi-task 3D understanding, including semantic, instance, panoptic, and text-referred segmentation.

Beyond segmentation, the high-quality, instance-aware 3D fields produced by SIU3R enable Versatile 3D Editing. Users can remove, replace, or re-color specific objects while maintaining perfect 3D consistency across views.

Limitations and Future Work

While SIU3R sets a new benchmark, challenges remain. The authors note that training on larger, more diverse datasets could enhance the model's "zero-shot" capabilities in complex or outdoor environments. Additionally, improving the quality of ground-truth labels—which are currently noisy in datasets like ScanNet—could further push the boundaries of segmentation accuracy. Finally, while the model handles sparse views effectively, extremely large viewpoint changes still present challenges for cross-view consistency.

Summary updated

BibTeX
@inproceedings{xu2025siu,
  author = {Qi Xu and Dongxu Wei and Lingzhe Zhao and Wenpu Li and Zhangchi Huang and Shunping Ji and Peidong Liu},
  title = {{SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=GtImvTta8x},
}
Figure