ORBIT: An Object Property Reasoning Benchmark for Visual Inference Tasks

Abhishek Kolari, Mohammadhossein Khojasteh, Yifan Jiang, Floris den Hengst, Filip Ilievski

Introduction

While state-of-the-art Vision-Language Models (VLMs) like GPT-4o and Qwen2.5-VL can describe complex scenes with impressive detail, their ability to truly understand the underlying properties of objects remains a subject of debate. For instance, if a chair is removed from a room, can an AI correctly deduce the remaining number of "seatable" objects? This level of commonsense reasoning is fundamental to human interaction with the physical world but remains a significant challenge for artificial intelligence.

The OPTICS benchmark (Object Property Reasoning in Visual Inference Tasks) was developed to systematically evaluate how VLMs reason over object properties such as taxonomy, physics, and function. Initial findings reveal a substantial performance gap: while humans achieve approximately 74% accuracy on complex counting tasks, even the most advanced models struggle to exceed 40%, frequently failing at basic physical reasoning and counterfactual scenarios.

Addressing the Reasoning Gap in VQA

Current Visual Question Answering (VQA) benchmarks often conflate simple perception—such as detecting a "red ball"—with higher-level reasoning, like understanding that a "ball" is a "toy" or is "spherical." Most existing datasets lack a clear delineation between these subtasks, making it difficult to diagnose specific model failures.

Comparison of OPTICS with existing VQA benchmarks across question types, object properties, image types, and reasoning complexity.

Furthermore, models often rely on statistical shortcuts rather than true abstraction. A model might "count" objects correctly in a simple scene but fail when presented with a counterfactual question (e.g., "If two of these lamps were removed, how many objects that require electricity would remain?"). For AI to be effective in real-world applications, it must transition from descriptive labeling to a functional understanding of its environment.

The OPTICS Framework

The researchers developed a systematic framework based on three primary axes of complexity to probe the limits of visual reasoning:

  1. Object Property Dimensions: This includes Physical properties (shape, color, material), Taxonomic categories (biological or artifact classifications), Functional attributes (what an object does, e.g., "means of transportation"), and Relational aspects (how objects interact).
  2. Reasoning Complexity: Tasks range from Direct Recognition (atomic perception) to Property Inference (multi-step abstraction) and Counterfactual Reasoning (hypothetical scene changes).
  3. Image Diversity: To test generalization, the benchmark utilizes Photographic, Animated, and AI-Generated images across 26 distinct domains, such as kitchens, classrooms, and zoos.
The OPTICS framework categorizes visual reasoning tasks by object property, reasoning complexity, and image domain.

Dataset Construction

The benchmark consists of two primary components:

  • OPTICS-CNT: 1,080 multi-level, count-based questions paired with 360 images.
  • OPTICS-CMP: 2,122 comparison questions (e.g., "Which image has more mammals?") designed to evaluate relative judgment.

Unlike many automated datasets, OPTICS was curated with significant human oversight to ensure high-quality, non-ambiguous ground truths, minimizing the noise often found in purely machine-generated benchmarks.

Key Insights and Experimental Results

Evaluation of 12 state-of-the-art VLMs, including the Qwen2.5-VL and InternVL3 families, yielded several critical insights:

  • The "Unnatural Error" Gap: Humans typically fail due to semantic ambiguity (e.g., debating if a stool qualifies as a "chair"). In contrast, models frequently commit "unnatural errors," such as counting non-existent objects or hallucinating properties (e.g., identifying a glass object as "metal").
  • The Counterfactual Challenge: Performance drops significantly on "what if" questions. This suggests that models are primarily performing pattern matching on pixels rather than building a robust mental model of the scene.
  • Domain Sensitivity: Models performed best on AI-generated and animated images, which tend to be "cleaner." They struggled most with photographic images, where occlusions and complex lighting conditions complicate visual grounding.
  • Scaling Limitations: While larger models generally outperformed smaller ones, the performance gains from increasing parameter counts were marginal compared to the massive gap between AI and human-level commonsense reasoning. The top-performing model, Qwen2.5-VL 32B, achieved only 39.9% micro-accuracy on OPTICS-CNT, far below the human average of 73.9%.

Conclusion and Future Directions

The OPTICS benchmark highlights that scaling alone may not be sufficient to bridge the gap in functional visual reasoning. High-quality reasoning benchmarks currently require intensive human annotation, which is a significant bottleneck. Future research may focus on developing "specialized reasoners"—models fine-tuned specifically for spatial and functional logic—and more sophisticated automated pipelines for generating commonsense questions. For now, OPTICS serves as a vital tool for measuring progress toward truly "intelligent" computer vision.

Summary updated

Abstract

Inspired by human categorization, object property reasoning involves identifying and recognizing low-level details and higher-level abstractions. While current visual question answering (VQA) studies consider multiple object properties, such as size, they typically blend perception and reasoning and lack representativeness in terms of reasoning and image categories, making it unclear whether and how vision-language models (VLMs) abstract and reason over depicted objects. To this end, we introduce a systematic evaluation framework comprising images of three representative types, three reasoning levels of increasing complexity, and four object property dimensions, informed by prior work on common sense. We develop a procedure to instantiate this framework in two VQA object reasoning benchmarks: OPTICS-CNT, comprising 360 images paired with 1,080 multi-level, count-based questions, and OPTICS-CMP, with 2.1k comparison questions. Experiments with 12 state-of-the-art VLMs in zero-shot settings reveal significant limitations relative to humans, with the best-performing model achieving below 40% counting and 70% comparison accuracy. VLMs struggle particularly with photographic images, counterfactual reasoning, physical and functional properties, and higher counts. We make the OPTICS benchmark data and code available to support future work on scalable benchmarking methods, generalized annotation guidelines, and advanced reasoning VLMs.
BibTeX
@misc{kolari2025orbit,
  author = {Abhishek Kolari and Mohammadhossein Khojasteh and Yifan Jiang and Floris den Hengst and Filip Ilievski},
  title = {{ORBIT: An Object Property Reasoning Benchmark for Visual Inference Tasks}},
  year = {2025},
  eprint = {2508.10956},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.10956},
}
Figure