Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention

Weitai Kang, Mengxue Qu, Jyoti Kini, Yunchao Wei, Mubarak Shah, Yan Yan

Abstract

In real-life scenarios, humans seek out objects in the 3D world to fulfill daily needs or intentions. This paper introduces 3D Intention Grounding (3D-IG), a novel task in 3D object detection that identifies targets based on abstract human intentions (e.g., "I want something to support my back") rather than explicit physical descriptions. To support this task, we present the Intent3D dataset, containing 44,990 intention texts across 1,042 scenes, and IntentNet, a framework utilizing cascaded adaptive learning to bridge the gap between abstract language and 3D geometry.

Introduction: From Description to Intention

Traditional 3D Visual Grounding (3D-VG) relies on explicit references, such as "the red pillow on the couch." This assumes a human has already observed the scene and identified the target. In contrast, 3D Intention Grounding (3D-IG) requires an AI agent to automatically observe, reason, and detect targets solely based on human intention.

Comparison between 3D Visual Grounding and 3D Intention Grounding.

As shown in Figure 1, while 3D-VG matches a description to an object, 3D-IG requires the system to understand that the intention to "relieve back pressure" logically maps to a "pillow." This shift moves the challenge from simple referential matching to complex intentional reasoning in 3D space.

The Intent3D Dataset

To facilitate research in this new domain, we developed the Intent3D dataset. It is built upon 1,042 scenes from the well-known ScanNet dataset and covers 209 fine-grained object classes.

The four-step pipeline for constructing the Intent3D dataset.

The construction process, illustrated in Figure 4, involves four key steps:

  1. Scene Graph Construction: Mapping the spatial relationships of objects in ScanNet.
  2. Object Selection: Identifying common, non-trivial, and unambiguous objects.
  3. Text Generation: Using GPT-4 to generate intention texts. Crucially, "negative constraints" were applied to prevent the model from mentioning object names, categories, or locations, ensuring the task requires genuine reasoning.
  4. Data Cleaning: Removing garbled text and regenerating ambiguous or repeated cases.

IntentNet: Reasoning through Action

Our proposed framework, IntentNet, is designed to handle the abstract nature of intentions by focusing on verbs—the "actions" that link intentions to objects.

Multimodal Architecture

IntentNet utilizes PointNet++ for 3D geometry and RoBERTa for text processing. A pretrained GroupFree detector suggests candidate object boxes, which are then matched against text queries. Unlike standard models that prioritize nouns, IntentNet aligns query features with the verbs in the intention text to identify potential object candidates.

Cascaded Adaptive Learning (CAL)

Training a model for intention grounding is challenging due to the logical hierarchy of the task. We introduce Cascaded Adaptive Learning (CAL), which adaptively scales loss functions based on a priority logic: the model must first understand the verb (action), then the object semantics, and finally the precise bounding box regression. This ensures stable training by mastering foundational concepts before tackling complex spatial localization.

Key Results and Insights

IntentNet sets a new state-of-the-art on the Intent3D benchmark, significantly outperforming existing 3D-VG models and large foundation models.

  • Superior Accuracy: IntentNet achieved a [email protected] of 58.92%, compared to 47.86% for the best baseline (BUTD-DETR).
  • Precision Gains: It nearly doubled the Average Precision ([email protected]) of its nearest competitor (27.60% vs. 13.46%).
  • The Multi-Instance Challenge: Intent3D often involves multiple targets (e.g., "I want to seat my colleagues" referring to four chairs). IntentNet's architecture is specifically tuned for this multi-instance detection problem.

Conclusion and Future Work

Intent3D represents a significant step toward more natural human-robot interaction. While IntentNet demonstrates strong reasoning capabilities, challenges remain in handling complex, multi-step intentions and overcoming dependencies on initial 3D detectors. Future work will explore integrating more powerful 3D vision-language foundation models to further bridge the gap between human thought and robotic action.

Summary updated

Abstract

In real-life scenarios, humans seek out objects in the 3D world to fulfill their daily needs or intentions. This inspires us to introduce 3D intention grounding, a new task in 3D object detection employing RGB-D, based on human intention, such as "I want something to support my back". Closely related, 3D visual grounding focuses on understanding human reference. To achieve detection based on human intention, it relies on humans to observe the scene, reason out the target that aligns with their intention ("pillow" in this case), and finally provide a reference to the AI system, such as "A pillow on the couch". Instead, 3D intention grounding challenges AI agents to automatically observe, reason and detect the desired target solely based on human intention. To tackle this challenge, we introduce the new Intent3D dataset, consisting of 44,990 intention texts associated with 209 fine-grained classes from 1,042 scenes of the ScanNet dataset. We also establish several baselines based on different language-based 3D object detection models on our benchmark. Finally, we propose IntentNet, our unique approach, designed to tackle this intention-based detection problem. It focuses on three key aspects: intention understanding, reasoning to identify object candidates, and cascaded adaptive learning that leverages the intrinsic priority logic of different losses for multiple objective optimization.
BibTeX
@inproceedings{kang2025intent,
  author = {Weitai Kang and Mengxue Qu and Jyoti Kini and Yunchao Wei and Mubarak Shah and Yan Yan},
  title = {{Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=5GgjiRzYp3},
}
Figure