RIO: A Benchmark for Reasoning Intention-Oriented Objects in Open Environments

Mengxue Qu, Yu Wu, Wu Liu, Xiaodan Liang, Jingkuan Song, Yao Zhao, Yunchao Wei

TL;DR

The RIO (Reasoning Intention-Oriented Objects) benchmark shifts the focus of computer vision from identifying what an object is to reasoning about what it can do for a user. By providing a large-scale dataset of 130,585 intention-object pairs described in natural language, the authors challenge current models to locate objects based on complex human needs (e.g., "something to lounge on while reading") rather than simple category labels.

What Problem Does This Paper Address?

Traditional object detection is "category-oriented"—models are trained to find a "chair" or a "knife." However, humans usually interact with the world based on "intentions." If you want to "cut a cake," you instinctively look for a sharp edge; if you want to "rest," you look for a flat, comfortable surface.

While the concept of "Affordance" (the potential actions an object offers) has been studied, it is often limited to simple verb-noun mappings (e.g., "sit-on"). This approach fails in open environments where a single verb like "play" could apply to a piano or a frisbee, and where the best object for a task depends heavily on the scene's context.

Comparison of object detection tasks, highlighting the shift from simple affordance and referring expressions to complex Intention-Oriented Object Detection (IOOD).

The authors identify a critical gap: we lack a benchmark that tests a model's ability to reason through complex, context-aware, natural language descriptions of human intent to find the right tool for the job.

Methods

The authors developed a multi-stage pipeline to create the RIO dataset:

  1. Image Selection: They filtered the COCO dataset for complex scenes containing at least three different object categories.
  2. Intention Generation via LLM: To move beyond simple verbs, they used ChatGPT (GPT-3.5). By providing the LLM with image captions and object lists, they prompted it to generate descriptive sentences about how a human might use a specific object in that specific scene.
  3. Human Refinement: Annotators reviewed these drafts, filtering out irrational descriptions and ensuring the "best match" between the intention and the object.
  4. Sentence Modification: To ensure the task requires true reasoning, they automatically stripped category names from the descriptions (e.g., "You can use the sofa to relax" becomes "Something you can use to relax").

The paper establishes two primary tasks based on this data: Intention-Oriented Object Detection (IOOD) and Intention-Oriented Instance Segmentation (IOIS).

The RIO Dataset

The primary contribution of the paper is the RIO Dataset itself:

  • Source: Built on top of COCO 2014.
  • Scale: 40,214 images and 130,585 intention-object pairs.
  • Diversity: Includes over 100 affordance types described in full sentences.
  • Splits: Features a "Common" set for standard training/testing and an "Uncommon" set specifically designed to test few-shot reasoning on long-tail, creative uses of objects (e.g., using a spoon to cut soft strawberries).

Key Insights and Results

  • Context is King: In RIO, the same intention might point to different objects depending on the scene. "Something to lie down on" might be a bed in a house, but a bench or even grass in a park.
  • End-to-End vs. Hybrid Models: The authors compared "Detector-Reasoner" hybrids (using GroundingDINO to find objects and ChatGPT to pick the right one) against end-to-end Vision-Language models (like MDETR and TOIST). They found that end-to-end models generally perform better because they align visual and textual features directly rather than relying on a fixed set of predicted labels.
  • The Performance Gap: While models like TOIST achieved respectable results on the "Common" test set (~49.05 [email protected]), performance plummeted on the "Uncommon" set (~21.96 [email protected]). This highlights that current models struggle with creative or non-standard object utility.
  • Zero-Shot Limitations: Hybrid models (GroundingDINO + ChatGPT) struggled with category misclassification, proving that simply "chaining" a detector and an LLM isn't enough for robust intention reasoning.

Limitations and Future Work

The massive drop in performance on the "Uncommon" set shows that models still rely heavily on common category-intention correlations rather than true functional reasoning. Furthermore, human intent is inherently subjective; while the authors used multiple quality checks, what one person considers "suitable for resting" might differ from another.

The authors suggest that future work should focus on incorporating more explicit environmental reasoning and external knowledge bases to help models understand the physical properties of objects beyond their labels.

Summary updated

Abstract

Intention-oriented object detection aims to detect desired objects based on specific intentions or requirements. For instance, when we desire to "lie down and rest", we instinctively seek out a suitable option such as a "bed" or a "sofa" that can fulfill our needs. Previous work in this area is limited either by the number of intention descriptions or by the affordance vocabulary available for intention objects. These limitations make it challenging to handle intentions in open environments effectively. To facilitate this research, we construct a comprehensive dataset called Reasoning Intention-Oriented Objects (RIO). In particular, RIO is specifically designed to incorporate diverse real-world scenarios and a wide range of object categories. It offers the following key features: 1) intention descriptions in RIO are represented as natural sentences rather than a mere word or verb phrase, making them more practical and meaningful; 2) the intention descriptions are contextually relevant to the scene, enabling a broader range of potential functionalities associated with the objects; 3) the dataset comprises a total of 40,214 images and 130,585 intention-object pairs. With the proposed RIO, we evaluate the ability of some existing models to reason intention-oriented objects in open environments.
BibTeX
@inproceedings{qu2023rio,
  author = {Mengxue Qu and Yu Wu and Wu Liu and Xiaodan Liang and Jingkuan Song and Yao Zhao and Yunchao Wei},
  title = {{RIO: A Benchmark for Reasoning Intention-Oriented Objects in Open Environments}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=c5DUGninMz},
}
Figure