TL;DR
This paper introduces 3D Intention Grounding (3D-IG), a novel task where an AI agent must directly locate objects in a 3D scene based purely on implicit human intentions (e.g., "I want something to support my back") rather than explicit descriptions. To benchmark this challenge, the authors release the Intent3D dataset and propose IntentNet, a specialized architecture that leverages verb-object alignment and a prioritized, cascaded loss optimization strategy to significantly outperform existing 3D vision-language models.
What Problem Does This Paper Address?
Traditional 3D Visual Grounding (3D-VG) models excel at finding objects when given explicit, descriptive language—for example, "the blue pillow on the left side of the brown couch." However, in real-world robotics or assistive technologies, humans rarely communicate this way. Instead, we express our immediate needs or intentions.
To bridge this gap, the authors propose 3D Intention Grounding (3D-IG). As illustrated below, while traditional 3D-VG requires the human to observe the scene, reason out the target object, and describe its spatial attributes, 3D-IG shifts the entire burden of observation and reasoning onto the AI agent. The agent must ingest a raw 3D point cloud and a free-form intention statement, infer which object class satisfies that intention, and predict its exact 3D bounding box.
Methods
To tackle the unique challenges of 3D-IG, the authors developed IntentNet, an architecture designed to jointly optimize 3D perception and deep intention comprehension.
IntentNet consists of three core innovations:
- Candidate Box Matching: Unlike explicit grounding, intention language doesn't name the target object. IntentNet uses a pretrained 3D object detector (GroupFree) to generate candidate bounding boxes, which are then explicitly matched against target regions to ground the model's spatial attention.
- Verb-Object Alignment: Human intentions are fundamentally rooted in actions. IntentNet uses a three-step language-vision alignment process:
- Verb Position Prediction: Locates verbs within the text query using part-of-speech tagging.
- Query-Verb Contrastive Learning: Aligns the model's learned queries with the semantic meaning of the verbs.
- Modulated Query-Object Contrastive Learning: Modulates the queries by element-wise multiplication with verb tokens to predict and align with the corresponding object features.
- Cascaded Adaptive Learning: Because the model optimizes multiple complex objectives simultaneously, the authors introduce a logical priority chain for its loss functions ($L_{vPos} \rightarrow L_{vSem} \rightarrow L_{voSem} \rightarrow L_{box}$). Higher-priority losses (like finding the verb) are converted into adaptive scaling factors that dynamically adjust the weight of subsequent lower-priority losses (like bounding box regression) during training.
Datasets
The authors introduced the Intent3D dataset to serve as the benchmark for this new task.
- Scale: 44,990 free-form intention texts.
- Diversity: Covers 209 fine-grained object classes across 1,042 indoor scenes from the ScanNet dataset.
- Curation Pipeline: The dataset was generated using ChatGPT (GPT-4) with strict prompt constraints to prevent the model from leaking explicit object categories, locations, or attributes. The generated data underwent rigorous manual cleaning to filter out ambiguous cases, repeated sentences, and garbled text.
Key Insights and Ideas
- Verbs are the New Nouns: In standard visual grounding, nouns dictate the target. In intention grounding, verbs (e.g., "support", "hydrate", "dispose") carry the critical signal. IntentNet's explicit modulation of queries using verb tokens acknowledges that an object's identity is defined by its utility to the user.
- Multi-Instance Complexity: 3D-IG is inherently a multi-instance detection problem. An intention like "I want to present slides to my coworkers" might require the model to ground multiple "Monitor" instances within a single conference room, making it significantly harder than single-target 3D-VG.
- Hierarchical Optimization: Standard multi-task loss weighting often fails when tasks are sequentially dependent. By enforcing a logical hierarchy (you must understand the action before you can find the object, and you must find the object before you can draw its box), Cascaded Adaptive Learning stabilizes multi-objective training.
Key Results
The authors evaluated IntentNet against three classes of language-guided 3D object detectors: expert models (BUTD-DETR, EDA), a 3D foundation model (3D-VisTA), and an LLM-based 3D model (Chat-3D-v2).
- Validation Performance: IntentNet achieved a 58.34% [email protected] and 25.36% [email protected], outperforming the closest competitor (3D-VisTA) by 11.22% in accuracy and 5.43% in Average Precision.
- Test Performance: On the test set, IntentNet maintained its lead with 58.92% [email protected] and 27.60% [email protected].
- LLM Baselines: Large Language Model approaches like Chat-3D-v2 performed poorly on AP metrics (e.g., 2.58% [email protected] on Val), primarily due to severe hallucination issues and difficulties mapping 3D visual features into the LLM space.
Limitations and Future Work
Despite its strong performance, IntentNet exhibits specific failure modes:
- Multi-Action Distractors: The model struggles when an intention contains multiple verbs or secondary actions. For example, given the prompt "I need to cool down on hot days while sitting in the living room," the model can get distracted by the word "sitting" and incorrectly detect a couch instead of a fan.
- Detector Dependence: The framework relies heavily on the initial quality of candidate bounding boxes generated by the underlying 3D detector. If the detector misses an object entirely, IntentNet cannot ground it.
Future work will likely focus on end-to-end architectures that tightly couple intention reasoning directly into the 3D proposal generation phase.
Summary updated