TL;DR
This paper introduces Intention-driven Visual Grounding (IVG), a new task that requires AI agents to locate objects based on human intentions (e.g., "I'm tired and need to rest") rather than literal descriptions. To support this, the authors released IntentionVG, a massive egocentric dataset with 500k free-form intention expressions across multiple scenes, providing a new benchmark for more natural human-robot interaction.
What Problem Does This Paper Address?
Visual grounding (VG) is the task of finding an object in an image given a text prompt. Traditionally, this has relied on literal descriptions like "the back couch under the mirror." However, in real-world robotics, humans rarely speak this way. We express intentions: "I'm feeling kind of tired now and really need to rest for a while."
Existing VG datasets fail to capture this nuance for three main reasons:
- Literal Bias: They assume descriptions are explicit and detailed, which impedes practical deployment where users prefer intention-based shorthand.
- Perspective Gap: Most data is collected from a third-person perspective, while embodied agents (like robots) operate from an egocentric (first-person) view.
- Single-Scene Limitation: They usually focus on one static image, whereas a real agent must search across multiple rooms or dynamic environments to find what a user needs.
Methods
The authors propose a two-stage pipeline for the IVG task:
- Intention Interpretation & Multi-Scene Perception: The agent receives a set of candidate scenes (images of different rooms) and a human intention query. It must first identify which scene most likely contains the object that satisfies the user's need.
- Visual Grounding: Once the correct scene is selected, the agent must localize the specific target object within that image, returning a bounding box and category tag.
The researchers evaluated several baseline architectures, ranging from "integrated" models (combining a scene perceiver like EVA-CLIP with a grounding model like Shikra) to "end-to-end" models based on Qwen-VL that handle both stages simultaneously through instruction tuning.
Datasets
The core contribution is the IntentionVG dataset. It is built upon the EgoObjects dataset to ensure a realistic egocentric perspective.
- Scale: 100K images and nearly 500K intention-based expressions.
- Diversity: Covers 1,096 object categories across 10 indoor scene types (e.g., kitchen, study, bathroom).
- Free-form Expressions: Unlike previous datasets that use rigid templates (e.g., "Something to [verb]"), IntentionVG uses GPT-4 to generate natural, diverse human-like expressions, which were then manually filtered for quality.
Key Insights and Ideas
- Intent vs. Literal: Models that excel at literal grounding (REC) often fail at intention-driven grounding. Understanding why a user wants something is fundamentally different from understanding what it looks like.
- The Multi-Scene Hurdle: Searching across multiple images significantly increases task difficulty. Models often get "confused" if too many scenes are presented at once during training; the authors found that 5 scenes is an optimal "sweet spot" for training stability.
- Supervision Matters: The best performance comes from supervising the model with not just the bounding box, but also the scene category and object tag. This helps the model build explicit associations between high-level intent and specific visual contexts.
- Egocentric Advantage: By using first-person data, the dataset aligns more closely with the actual visual input an embodied agent would receive, making it a more robust benchmark for robotics.
Key Results
- Fine-tuning is Essential: Zero-shot SOTA models (like Shikra or Qwen-VL) performed poorly on IVG without specific training, struggling to bridge the gap between intent and location.
- SOTA Performance: The fine-tuned end-to-end Qwen-VL model achieved the best results, with a [email protected] of 50.58% in single-scene settings and 39.30% in the more challenging multi-scene setting.
- Data Quality: Ablation studies showed that using free-form expressions led to significantly better performance than using simple object tags or rigid templates, proving that linguistic diversity helps models learn more robust representations.
Limitations and Future Work
The authors identify two main areas for improvement:
- Output Granularity: Currently, the models only produce bounding boxes. Real-world manipulation often requires pixel-perfect segmentation masks. The authors suggest integrating visual foundation models like SAM (Segment Anything Model) in future iterations.
- Model Capacity: While the baselines are strong, there is still a significant performance gap between single-scene and multi-scene grounding, suggesting that more advanced architectures for long-range visual reasoning and navigation are needed.
Summary updated