TL;DR
The paper introduces Detailed 3D Referring Expression Segmentation (3D-DRES), a novel task that moves beyond traditional sentence-level visual grounding by mapping individual noun phrases within a description to their exact 3D shapes. To support this, the authors release DetailRefer, a large-scale dataset featuring dense phrase-to-instance annotations, alongside DetailBase, a baseline model demonstrating that learning fine-grained phrase context significantly boosts overall 3D spatial reasoning.
What Problem Does This Paper Address?
Existing 3D visual grounding methods operate under a "single-unit assumption." Whether localizing an object with a bounding box (3D-REC) or segmenting it at the point-cloud level (3D-RES), these systems focus entirely on a single target object described by a sentence.
For example, given the prompt "The trash can next to the thin table under the TV," traditional models only output a mask for the "trash can." They completely ignore whether the network actually understands the anchor objects ("table" and "TV") used to find it. This sentence-level approach fails to leverage the rich, compositional context of natural language and limits real-world applications—like an embodied robot trying to follow multi-step instructions involving multiple objects.
To bridge this gap, the authors propose 3D-DRES, requiring models to segment every noun phrase mentioned in a description individually.
Methods
Because existing 3D-RES architectures cannot output multiple distinct masks for specific tokens, the authors developed DetailBase, a streamlined and extensible baseline architecture:
- Visual Feature Extraction: A Sparse 3D U-Net extracts point-level features from the 3D scene. To manage computational complexity, the point cloud undergoes unsupervised over-segmentation into superpoints, which are then aggregated via superpoint pooling.
- Textual Feature Extraction: A pre-trained MPNet processes the text. Instead of using a global sentence embedding, token features at the exact positions of the target nouns are extracted to initialize separate queries ($Q_0$).
- Multi-modal Fusion Decoder: The queries pass through a series of decoder layers utilizing Cross-Attention (to inject visual scene features), Self-Attention (to capture intra-sentence relationships), and Feed-Forward Networks.
- Dual-Mode Mask Generation: The network computes the affinity between the final query outputs and the superpoint features, generating distinct point-level masks for each phrase. It can also perform traditional sentence-level segmentation by using the
[CLS]token query.
Datasets
The authors introduced DetailRefer, a new dataset built by expanding the ScanRefer benchmark using a pipeline of Large Language Models (LLMs) and roughly 600 hours of manual verification.
- Scale: 54,432 descriptions spanning 11,054 distinct indoor objects from the ScanNet dataset.
- Complexity: Features an average text length of 24.9 tokens—significantly longer than existing datasets.
- Density: Implements a phrase-segmentation format resulting in an unprecedented average of 2.9 masks per text description. It also includes strategically integrated "Long" texts (>50 tokens) and "Complex" samples requiring four or more distinct object masks.
Key Insights and Ideas
- Synergistic Tasks: Phrase-level and sentence-level segmentation are not mutually exclusive; they are deeply complementary.
- Contextual Grounding Boosts Target Grounding: Forcing a model to explicitly ground intermediate context objects (like the table or TV) improves its internal spatial reasoning. This structural language understanding directly translates to superior performance on traditional single-target tasks.
- One-Stage Superiority: The authors adapted a two-stage domain baseline (PNG) that first generates instance proposals and then matches them to text. However, it significantly underperformed compared to one-stage joint-attention approaches, highlighting that tight, early multi-modal coupling is vital for fine-grained 3D grounding.
Key Results
- Baseline Performance: On the DetailRefer test set, DetailBase achieved a phrase-level mean Intersection-over-Union (mIoU) of 55.7%, outperforming adapted versions of PNG (40.4%) and 3D-STMN (52.5%).
- The Joint-Training Dividend: When DetailBase was jointly trained on both the traditional ScanRefer dataset and the new DetailRefer dataset, its performance on the traditional 3D-RES benchmark improved from 44.0% to 46.8% mIoU. Similarly, the adapted 3D-STMN model saw a 3.2-point performance boost from joint training.
Limitations and Future Work
Despite its strong performance, the authors note several open challenges for the 3D-DRES task:
- Lack of Instance-Level Awareness: Because DetailBase operates strictly at the superpoint level, it lacks top-down object instance perception. This makes it difficult to distinguish between multiple nearby objects of the exact same category.
- Complex Syntax: Highly intricate, long sentence structures still challenge the cross-attention layers' ability to correctly map linguistic dependencies to 3D space.
Future research directions will focus on effectively incorporating instance-level cues into single-stage grounding architectures.
Summary updated