Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions

Wenxuan Wang, Yisi Zhang, Xingjian He, Yichen Yan, Zijia Zhao, Xinlong Wang, Jing Liu

TL;DR

This paper introduces Intention-driven Visual Grounding (IVG), a new task that requires AI agents to locate objects based on human intentions (e.g., "I'm tired and need to rest") rather than literal descriptions. To support this, the authors released IntentionVG, a massive egocentric dataset with 500k free-form intention expressions across multiple scenes, providing a new benchmark for more natural human-robot interaction.

What Problem Does This Paper Address?

Visual grounding (VG) is the task of finding an object in an image given a text prompt. Traditionally, this has relied on literal descriptions like "the back couch under the mirror." However, in real-world robotics, humans rarely speak this way. We express intentions: "I'm feeling kind of tired now and really need to rest for a while."

Existing VG datasets fail to capture this nuance for three main reasons:

  1. Literal Bias: They assume descriptions are explicit and detailed, which impedes practical deployment where users prefer intention-based shorthand.
  2. Perspective Gap: Most data is collected from a third-person perspective, while embodied agents (like robots) operate from an egocentric (first-person) view.
  3. Single-Scene Limitation: They usually focus on one static image, whereas a real agent must search across multiple rooms or dynamic environments to find what a user needs.
Figure 1: Task Comparison between Affordance Detection (AD), Task-Driven Object Detection (TDOD), Referring Expression Comprehension (REC), Intention-Oriented Object Detection (IOOD) and Intention-driven VG (IVG).

Methods

The authors propose a two-stage pipeline for the IVG task:

  1. Intention Interpretation & Multi-Scene Perception: The agent receives a set of candidate scenes (images of different rooms) and a human intention query. It must first identify which scene most likely contains the object that satisfies the user's need.
  2. Visual Grounding: Once the correct scene is selected, the agent must localize the specific target object within that image, returning a bounding box and category tag.
Figure 2: The illustration about the overall pipeline of our intention-driven visual grounding task, which mainly comprises intention interpretation, multi-scene perception and the subsequent visual grounding.

The researchers evaluated several baseline architectures, ranging from "integrated" models (combining a scene perceiver like EVA-CLIP with a grounding model like Shikra) to "end-to-end" models based on Qwen-VL that handle both stages simultaneously through instruction tuning.

Datasets

The core contribution is the IntentionVG dataset. It is built upon the EgoObjects dataset to ensure a realistic egocentric perspective.

  • Scale: 100K images and nearly 500K intention-based expressions.
  • Diversity: Covers 1,096 object categories across 10 indoor scene types (e.g., kitchen, study, bathroom).
  • Free-form Expressions: Unlike previous datasets that use rigid templates (e.g., "Something to [verb]"), IntentionVG uses GPT-4 to generate natural, diverse human-like expressions, which were then manually filtered for quality.
Figure 3: The illustration of data collection engine for IntentionVG. We start by inheriting EgoObjects (Zhu et al., 2023a) data and conduct scene category labeling for each image. Then we feed GPT-4 with V-L input to generate the draft of intention-driven response. At last, we conduct data filtering by manually selecting the well matched expression-bounding box (bbox) pairs.

Key Insights and Ideas

  • Intent vs. Literal: Models that excel at literal grounding (REC) often fail at intention-driven grounding. Understanding why a user wants something is fundamentally different from understanding what it looks like.
  • The Multi-Scene Hurdle: Searching across multiple images significantly increases task difficulty. Models often get "confused" if too many scenes are presented at once during training; the authors found that 5 scenes is an optimal "sweet spot" for training stability.
  • Supervision Matters: The best performance comes from supervising the model with not just the bounding box, but also the scene category and object tag. This helps the model build explicit associations between high-level intent and specific visual contexts.
  • Egocentric Advantage: By using first-person data, the dataset aligns more closely with the actual visual input an embodied agent would receive, making it a more robust benchmark for robotics.

Key Results

  • Fine-tuning is Essential: Zero-shot SOTA models (like Shikra or Qwen-VL) performed poorly on IVG without specific training, struggling to bridge the gap between intent and location.
  • SOTA Performance: The fine-tuned end-to-end Qwen-VL model achieved the best results, with a [email protected] of 50.58% in single-scene settings and 39.30% in the more challenging multi-scene setting.
  • Data Quality: Ablation studies showed that using free-form expressions led to significantly better performance than using simple object tags or rigid templates, proving that linguistic diversity helps models learn more robust representations.
Figure 4: IntentionVG dataset statistics. (a) the number of referring expressions per object’s category in the log scale. (b) the word cloud highlights the head categories.

Limitations and Future Work

The authors identify two main areas for improvement:

  1. Output Granularity: Currently, the models only produce bounding boxes. Real-world manipulation often requires pixel-perfect segmentation masks. The authors suggest integrating visual foundation models like SAM (Segment Anything Model) in future iterations.
  2. Model Capacity: While the baselines are strong, there is still a significant performance gap between single-scene and multi-scene grounding, suggesting that more advanced architectures for long-range visual reasoning and navigation are needed.

Summary updated

Abstract

Visual grounding (VG) aims at locating the foreground entities that match the given natural language expressions. Previous datasets and methods for classic VG task mainly rely on the prior assumption that the given expression must literally refer to the target object, which greatly impedes the practical deployment of agents in real-world scenarios. Since users usually prefer to provide intention-based expression for the desired object instead of covering all the details, it is necessary for the agents to interpret the intention-driven instructions. Thus, in this work, we take a step further to the intention-driven visual-language (V-L) understanding. To promote classic VG towards human intention interpretation, we propose a new intention-driven visual grounding (IVG) task and build a large-scale IVG dataset termed IntentionVG with free-form intention expressions. Considering that practical agents need to move and find specific targets among various scenarios to realize the grounding task, our IVG task and IntentionVG dataset have taken the crucial properties of both multi-scenario perception and egocentric view into consideration. Besides, various types of models are set up as the baselines to realize our IVG task. Extensive experiments on our IntentionVG dataset and baselines demonstrate the necessity and efficacy of our method for the V-L field. To foster future research in this direction, our newly built dataset and baselines will be publicly available at https://github.com/Rubics-Xuan/IVG.
BibTeX
@inproceedings{wang2024beyond,
  author = {Wenxuan Wang and Yisi Zhang and Xingjian He and Yichen Yan and Zijia Zhao and Xinlong Wang and Jing Liu},
  title = {{Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.43},
}
Figure