Abstract: The recent development in multimodal learning has greatly advanced the research in 3D scene understanding in various real-world tasks such as embodied AI. However, most existing work shares two typical constraints: 1) they are short of reasoning ability for interaction and interpretation of human intension and 2) they focus on scenarios with single-category objects only which leads to over-simplified textual descriptions due to the negligence of multi-object scenarios and spatial relations among objects. We bridge the research gaps by proposing a 3D reasoning segmentation task for multiple objects in scenes. The task allows producing 3D segmentation masks and detailed textual explanations as enriched by 3D spatial relations among objects. To this end, we create ReasonSeg3D, a large-scale and high-quality benchmark that integrates 3D spatial relations with generated question-answer pairs and 3D segmentation masks. In addition, we design MORE3D, a simple yet effective method that enables multi-object 3D reasoning segmentation with user questions and textual outputs. Extensive experiments show that MORE3D excels in reasoning and segmenting complex multi-object 3D scenes, and the created ReasonSeg3D offers a valuable platform for future exploration of 3D reasoning segmentation.
Introduction
While 2D image understanding has seen significant breakthroughs, 3D scene understanding has historically lagged in its ability to perform high-level reasoning. Most current 3D models are restricted to explicit tasks, such as segmenting a specific object when directly named (referring) or providing basic room descriptions (captioning). These models typically struggle with two major challenges: interpreting implicit human intent and navigating complex, multi-object environments.
To address these gaps, researchers have introduced MORE3D, a framework designed to understand complex 3D scenes and segment multiple objects based on implicit user needs. For instance, if a user asks, "How can I enhance my workspace productivity?" the system must reason that it should identify and segment the desk, chair, and bookshelf simultaneously.
The MORE3D Framework
The Multi-Object REasoning 3D network (MORE3D) employs a sophisticated architecture to bridge the gap between raw 3D data and linguistic reasoning:
- 3D Encoding: The system processes raw point clouds using a 3D encoder to extract per-point features, which are then projected into a space compatible with a Large Language Model (LLM).
- Reasoning Engine: A fine-tuned LLaMA-7B model acts as the central processor. It interprets the 3D features alongside user queries to generate detailed, context-aware explanations.
- Multi-Object Segmentation: Unlike traditional models that output a single mask, MORE3D generates a specialized
<SEG>token for every relevant object identified in its textual explanation. - Object-Specific Embedding Extraction: Unique embeddings are extracted for each
<SEG>token and combined with the original 3D point features via a 3D Decoder. This allows the model to produce precise, individual segmentation masks for different object categories at once.
The ReasonSeg3D Benchmark
To support this new task, the researchers developed ReasonSeg3D, a large-scale benchmark that surpasses previous datasets in both scale and complexity. Sourced from ScanNetv2, the dataset includes 1,513 scenes and 20,113 question-answer pairs.
Key features of ReasonSeg3D include:
- Spatial Awareness: Ground-truth answers include explicit 3D spatial relations (e.g., "positioned on either side"), teaching the model the "why" behind its visual outputs.
- High-Quality Generation: The dataset was generated using GPT-4o, leveraging its visual reasoning capabilities to create practical, spatially-aware questions that go beyond simple text-only pipelines.
Key Insights and Results
Experimental results demonstrate that MORE3D sets a new state-of-the-art on the ReasonSeg3D benchmark, achieving a 30.19% cIoU (cumulative Intersection over Union), a significant improvement over the previous best of 25.12%.
A critical finding of the study is that spatial relations serve as a vital form of supervision. Requiring the model to explain where objects are relative to one another significantly improves visual segmentation accuracy. Furthermore, the authors found that using separate network branches for segmentation and classification—rather than a unified head—reduces the learning burden on the decoder, leading to better specialization.
Conclusion and Future Work
MORE3D represents a shift from "point and click" interactions toward more useful embodied AI capable of assisting humans in real-world tasks. While currently focused on indoor environments, future work will aim to extend these capabilities to complex outdoor scenes and dynamic environments. Additionally, optimizing the inference speed of large LLM-based 3D pipelines remains a priority for real-time applications in robotics and augmented reality.
Summary updated