A Study of Commonsense Reasoning over Visual Object Properties

Abhishek Kolari, Mohammadhossein Khojasteh, Yifan Jiang, Floris den Hengst, Filip Ilievski

TL;DR

Researchers have introduced OPTICS, a new evaluation framework designed to stress-test how Vision-Language Models (VLMs) reason about object properties like "function" and "physicality." The results are a wake-up call: even the best models (like Qwen2.5-VL) achieve less than 40% accuracy on complex counting tasks, struggling significantly with counterfactual scenarios and messy, real-world photographic images where humans still reign supreme.

What Problem Does This Paper Address?

While modern VLMs are impressive, we don't actually know if they understand the objects they see or if they are just very good at pattern matching. Most existing Visual Question Answering (VQA) benchmarks conflate simple perception (detecting a cat) with higher-level reasoning (knowing that a cat is a mammal or imagining what happens if the cat leaves the room).

The authors argue that current benchmarks lack a systematic way to separate these layers. Without this separation, it’s unclear whether a model fails because it didn't "see" the object or because it lacks the commonsense knowledge to reason about its properties (e.g., "Is this object breakable?").

Methods

The authors propose a multi-dimensional framework to categorize object reasoning across three axes:

  1. Object Property Dimensions: Physical (material/state), Taxonomic (biological/category), Functional (utility/use-case), and Relational (spatial/contextual).
  2. Reasoning Complexity:
    • Direct Recognition: Identifying what is there.
  3. Property Inference: Abstracting beyond the surface (e.g., "How many of these are means of transportation?").
  4. Counterfactual Reasoning: Reasoning about hypothetical changes (e.g., "If half the clocks were removed, how many would remain?").
  5. Image Types: Photographic (real world), Animated (stylized), and AI-Generated (synthetic).

To build the dataset, they used a "semi-automatic" pipeline: they prompted state-of-the-art MLLMs to generate candidate questions for 360 images, which were then rigorously refined, corrected, and verified by human annotators to ensure high quality and "common sense" validity.

Datasets

The paper introduces two new benchmarks:

  • OPTICS-CNT: 1,080 multi-level, count-based questions across 360 images.
  • OPTICS-CMP: 2.1k comparison questions (e.g., "Which image has more breakable objects?") derived from the counting data.

The images span 26 diverse domains, including kitchens, zoos, urban environments, and classrooms.

Key Insights and Ideas

  • The "Unnatural Error" Gap: One of the most striking findings is the type of mistakes models make. Humans mostly fail due to "semantic ambiguity" (e.g., "Does a stool count as furniture?"). Models, however, make "unnatural errors"—hallucinating objects that don't exist or failing to count objects in plain sight.
  • Synthetic Bias: Models perform significantly better on AI-generated and animated images than on real-world photographs. This suggests that the "clutter" and lighting complexity of the real world still baffle current vision systems.
  • Reasoning vs. Perception: Surprisingly, some top models actually performed better on counterfactual questions than on direct recognition. This suggests they might be relying on linguistic shortcuts or "lucky" guesses rather than a solid perceptual foundation.
  • The Scaling Wall: The authors tried to automate the creation of these datasets using existing large-scale data like Visual Genome, but found the automated versions were too noisy and lacked the nuance required for true commonsense testing.

Key Results

  • Human vs. Machine: There is a massive performance gap. Humans averaged 74% accuracy on the counting tasks, while the best-performing model (Qwen2.5-VL-32B) hit only 39.9%.
  • The 40/70 Rule: Across 12 state-of-the-art models, the best achieved below 40% on counting (OPTICS-CNT) and roughly 70% on comparisons (OPTICS-CMP).
  • Model Size Matters (Mostly): Within model families (like Qwen or InternVL), performance generally scaled with parameter count, but even the largest models struggled with functional and physical properties.
  • Inductive Bias: Older, non-instruction-tuned models (like BLIP-2) showed severe "inductive bias," often defaulting to the same answer (like "1" or "3") regardless of the image content.

Limitations and Future Work

The primary limitation is the cost of human annotation. Because models make "unnatural" mistakes, humans must remain in the loop to verify ground truth, making it difficult to scale OPTICS to millions of examples.

Future work suggested by the authors includes:

  • Developing specialized "reasoner" models that use scene graphs to separate perception from logic.
  • Improving the robustness of VLMs to "cluttered" photographic scenes.
  • Creating more sophisticated automated pipelines that can generate high-quality commonsense questions without constant human oversight.

Summary updated

Abstract

Inspired by human categorization, object property reasoning involves identifying and recognizing low-level details and higher-level abstractions. While current visual question answering (VQA) studies consider multiple object properties, such as size, they typically blend perception and reasoning and lack representativeness in terms of reasoning and image categories, making it unclear whether and how vision-language models (VLMs) abstract and reason over depicted objects. To this end, we introduce a systematic evaluation framework comprising images of three representative types, three reasoning levels of increasing complexity, and four object property dimensions, informed by prior work on common sense. We develop a procedure to instantiate this framework in two VQA object reasoning benchmarks: OPTICS-CNT, comprising 360 images paired with 1,080 multi-level, count-based questions, and OPTICS-CMP, with 2.1k comparison questions. Experiments with 12 state-of-the-art VLMs in zero-shot settings reveal significant limitations relative to humans, with the best-performing model achieving below 40% counting and 70% comparison accuracy. VLMs struggle particularly with photographic images, counterfactual reasoning, physical and functional properties, and higher counts. We make the OPTICS benchmark data and code available to support future work on scalable benchmarking methods, generalized annotation guidelines, and advanced reasoning VLMs.
BibTeX
@misc{kolari2025study,
  author = {Abhishek Kolari and Mohammadhossein Khojasteh and Yifan Jiang and Floris den Hengst and Filip Ilievski},
  title = {{A Study of Commonsense Reasoning over Visual Object Properties}},
  year = {2025},
  eprint = {2508.10956},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.10956},
}
Figure