Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention

Weitai Kang, Mengxue Qu, Jyoti Kini, Yunchao Wei, Mubarak Shah, Yan Yan

TL;DR

This paper introduces 3D Intention Grounding (3D-IG), a novel task where an AI agent must directly locate objects in a 3D scene based purely on implicit human intentions (e.g., "I want something to support my back") rather than explicit descriptions. To benchmark this challenge, the authors release the Intent3D dataset and propose IntentNet, a specialized architecture that leverages verb-object alignment and a prioritized, cascaded loss optimization strategy to significantly outperform existing 3D vision-language models.

What Problem Does This Paper Address?

Traditional 3D Visual Grounding (3D-VG) models excel at finding objects when given explicit, descriptive language—for example, "the blue pillow on the left side of the brown couch." However, in real-world robotics or assistive technologies, humans rarely communicate this way. Instead, we express our immediate needs or intentions.

To bridge this gap, the authors propose 3D Intention Grounding (3D-IG). As illustrated below, while traditional 3D-VG requires the human to observe the scene, reason out the target object, and describe its spatial attributes, 3D-IG shifts the entire burden of observation and reasoning onto the AI agent. The agent must ingest a raw 3D point cloud and a free-form intention statement, infer which object class satisfies that intention, and predict its exact 3D bounding box.

Figure 1: We introduce 3D intention grounding (right), a new task for detecting the object of interest using a 3D bounding box in a 3D scene, guided by human intention expressed in text (e.g., “I want something to support my back to relieve the pressure”). In contrast, the existing 3D visual grounding (left) relies on human reasoning and references for detection. The illustration clearly distinguishes that observation and reasoning are manually executed by human (left) and automated by AI (right).

Methods

To tackle the unique challenges of 3D-IG, the authors developed IntentNet, an architecture designed to jointly optimize 3D perception and deep intention comprehension.

The IntentNet Architecture overview.

IntentNet consists of three core innovations:

  1. Candidate Box Matching: Unlike explicit grounding, intention language doesn't name the target object. IntentNet uses a pretrained 3D object detector (GroupFree) to generate candidate bounding boxes, which are then explicitly matched against target regions to ground the model's spatial attention.
  2. Verb-Object Alignment: Human intentions are fundamentally rooted in actions. IntentNet uses a three-step language-vision alignment process:
  3. Verb Position Prediction: Locates verbs within the text query using part-of-speech tagging.
  4. Query-Verb Contrastive Learning: Aligns the model's learned queries with the semantic meaning of the verbs.
  5. Modulated Query-Object Contrastive Learning: Modulates the queries by element-wise multiplication with verb tokens to predict and align with the corresponding object features.
  6. Cascaded Adaptive Learning: Because the model optimizes multiple complex objectives simultaneously, the authors introduce a logical priority chain for its loss functions ($L_{vPos} \rightarrow L_{vSem} \rightarrow L_{voSem} \rightarrow L_{box}$). Higher-priority losses (like finding the verb) are converted into adaptive scaling factors that dynamically adjust the weight of subsequent lower-priority losses (like bounding box regression) during training.

Datasets

The authors introduced the Intent3D dataset to serve as the benchmark for this new task.

  • Scale: 44,990 free-form intention texts.
  • Diversity: Covers 209 fine-grained object classes across 1,042 indoor scenes from the ScanNet dataset.
  • Curation Pipeline: The dataset was generated using ChatGPT (GPT-4) with strict prompt constraints to prevent the model from leaking explicit object categories, locations, or attributes. The generated data underwent rigorous manual cleaning to filter out ambiguous cases, repeated sentences, and garbled text.
Figure 2: (Upper row) Flowchart for dataset construction. After constructing the scene graph, we select objects by three criteria: Common Object, Non-trivial Object, Unambiguous Object. We use ChatGPT to generate intention texts given the prompt we designed. Finally, we manually clean the data. (Lower row) Examples in our dataset for different number of targets and length of texts.

Key Insights and Ideas

  • Verbs are the New Nouns: In standard visual grounding, nouns dictate the target. In intention grounding, verbs (e.g., "support", "hydrate", "dispose") carry the critical signal. IntentNet's explicit modulation of queries using verb tokens acknowledges that an object's identity is defined by its utility to the user.
  • Multi-Instance Complexity: 3D-IG is inherently a multi-instance detection problem. An intention like "I want to present slides to my coworkers" might require the model to ground multiple "Monitor" instances within a single conference room, making it significantly harder than single-target 3D-VG.
  • Hierarchical Optimization: Standard multi-task loss weighting often fails when tasks are sequentially dependent. By enforcing a logical hierarchy (you must understand the action before you can find the object, and you must find the object before you can draw its box), Cascaded Adaptive Learning stabilizes multi-objective training.

Key Results

The authors evaluated IntentNet against three classes of language-guided 3D object detectors: expert models (BUTD-DETR, EDA), a 3D foundation model (3D-VisTA), and an LLM-based 3D model (Chat-3D-v2).

  • Validation Performance: IntentNet achieved a 58.34% [email protected] and 25.36% [email protected], outperforming the closest competitor (3D-VisTA) by 11.22% in accuracy and 5.43% in Average Precision.
  • Test Performance: On the test set, IntentNet maintained its lead with 58.92% [email protected] and 27.60% [email protected].
  • LLM Baselines: Large Language Model approaches like Chat-3D-v2 performed poorly on AP metrics (e.g., 2.58% [email protected] on Val), primarily due to severe hallucination issues and difficulties mapping 3D visual features into the LLM space.

Limitations and Future Work

Despite its strong performance, IntentNet exhibits specific failure modes:

  • Multi-Action Distractors: The model struggles when an intention contains multiple verbs or secondary actions. For example, given the prompt "I need to cool down on hot days while sitting in the living room," the model can get distracted by the word "sitting" and incorrectly detect a couch instead of a fan.
  • Detector Dependence: The framework relies heavily on the initial quality of candidate bounding boxes generated by the underlying 3D detector. If the detector misses an object entirely, IntentNet cannot ground it.

Future work will likely focus on end-to-end architectures that tightly couple intention reasoning directly into the 3D proposal generation phase.

Summary updated

Abstract

In real-life scenarios, humans seek out objects in the 3D world to fulfill their daily needs or intentions. This inspires us to introduce 3D intention grounding, a new task in 3D object detection employing RGB-D, based on human intention, such as "I want something to support my back". Closely related, 3D visual grounding focuses on understanding human reference. To achieve detection based on human intention, it relies on humans to observe the scene, reason out the target that aligns with their intention ("pillow" in this case), and finally provide a reference to the AI system, such as "A pillow on the couch". Instead, 3D intention grounding challenges AI agents to automatically observe, reason and detect the desired target solely based on human intention. To tackle this challenge, we introduce the new Intent3D dataset, consisting of 44,990 intention texts associated with 209 fine-grained classes from 1,042 scenes of the ScanNet dataset. We also establish several baselines based on different language-based 3D object detection models on our benchmark. Finally, we propose IntentNet, our unique approach, designed to tackle this intention-based detection problem. It focuses on three key aspects: intention understanding, reasoning to identify object candidates, and cascaded adaptive learning that leverages the intrinsic priority logic of different losses for multiple objective optimization. Project Page: https://weitaikang.github.io/Intent3D-webpage/
BibTeX
@inproceedings{kang2025intent,
  author = {Weitai Kang and Mengxue Qu and Jyoti Kini and Yunchao Wei and Mubarak Shah and Yan Yan},
  title = {{Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=5GgjiRzYp3},
}
Figure