Abstract
In real-life scenarios, humans seek out objects in the 3D world to fulfill daily needs or intentions. This paper introduces 3D Intention Grounding (3D-IG), a novel task in 3D object detection that identifies targets based on abstract human intentions (e.g., "I want something to support my back") rather than explicit physical descriptions. To support this task, we present the Intent3D dataset, containing 44,990 intention texts across 1,042 scenes, and IntentNet, a framework utilizing cascaded adaptive learning to bridge the gap between abstract language and 3D geometry.
Introduction: From Description to Intention
Traditional 3D Visual Grounding (3D-VG) relies on explicit references, such as "the red pillow on the couch." This assumes a human has already observed the scene and identified the target. In contrast, 3D Intention Grounding (3D-IG) requires an AI agent to automatically observe, reason, and detect targets solely based on human intention.
As shown in Figure 1, while 3D-VG matches a description to an object, 3D-IG requires the system to understand that the intention to "relieve back pressure" logically maps to a "pillow." This shift moves the challenge from simple referential matching to complex intentional reasoning in 3D space.
The Intent3D Dataset
To facilitate research in this new domain, we developed the Intent3D dataset. It is built upon 1,042 scenes from the well-known ScanNet dataset and covers 209 fine-grained object classes.
The construction process, illustrated in Figure 4, involves four key steps:
- Scene Graph Construction: Mapping the spatial relationships of objects in ScanNet.
- Object Selection: Identifying common, non-trivial, and unambiguous objects.
- Text Generation: Using GPT-4 to generate intention texts. Crucially, "negative constraints" were applied to prevent the model from mentioning object names, categories, or locations, ensuring the task requires genuine reasoning.
- Data Cleaning: Removing garbled text and regenerating ambiguous or repeated cases.
IntentNet: Reasoning through Action
Our proposed framework, IntentNet, is designed to handle the abstract nature of intentions by focusing on verbs—the "actions" that link intentions to objects.
Multimodal Architecture
IntentNet utilizes PointNet++ for 3D geometry and RoBERTa for text processing. A pretrained GroupFree detector suggests candidate object boxes, which are then matched against text queries. Unlike standard models that prioritize nouns, IntentNet aligns query features with the verbs in the intention text to identify potential object candidates.
Cascaded Adaptive Learning (CAL)
Training a model for intention grounding is challenging due to the logical hierarchy of the task. We introduce Cascaded Adaptive Learning (CAL), which adaptively scales loss functions based on a priority logic: the model must first understand the verb (action), then the object semantics, and finally the precise bounding box regression. This ensures stable training by mastering foundational concepts before tackling complex spatial localization.
Key Results and Insights
IntentNet sets a new state-of-the-art on the Intent3D benchmark, significantly outperforming existing 3D-VG models and large foundation models.
- Superior Accuracy: IntentNet achieved a [email protected] of 58.92%, compared to 47.86% for the best baseline (BUTD-DETR).
- Precision Gains: It nearly doubled the Average Precision ([email protected]) of its nearest competitor (27.60% vs. 13.46%).
- The Multi-Instance Challenge: Intent3D often involves multiple targets (e.g., "I want to seat my colleagues" referring to four chairs). IntentNet's architecture is specifically tuned for this multi-instance detection problem.
Conclusion and Future Work
Intent3D represents a significant step toward more natural human-robot interaction. While IntentNet demonstrates strong reasoning capabilities, challenges remain in handling complex, multi-step intentions and overcoming dependencies on initial 3D detectors. Future work will explore integrating more powerful 3D vision-language foundation models to further bridge the gap between human thought and robotic action.
Summary updated