Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models

Zahra Babaiee, Peyman Kiasari, Daniela Rus, Radu Grosu

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in object recognition and visual question answering. However, the ability of these models to perform "visual conceptualization"—recognizing abstract concepts regardless of their visual representation—remains largely untested. This paper introduces the Visual Graph Arena (VGA), a comprehensive benchmark designed to evaluate representation-invariant reasoning. Our findings reveal a significant "reasoning gap" between human performance and state-of-the-art AI, suggesting that current models rely on superficial pattern matching rather than structural understanding.

1. Introduction

A fundamental aspect of human intelligence is the ability to perceive the "forest for the trees." Humans can identify a chemical structure, a logic circuit, or a social network regardless of how the data is visually rendered. This ability, known as visual conceptualization, requires an agent to extract the underlying structural truth (invariants) from varying visual layouts. While MLLMs like GPT-4o and Claude 3.5 show high proficiency in standard benchmarks, their ability to generalize across different visual styles of the same data remains a critical bottleneck for achieving Artificial General Intelligence (AGI).

2. The Visual Graph Arena (VGA) Framework

To rigorously test visual conceptualization, we developed the Visual Graph Arena (VGA). Graphs serve as an ideal testbed because a graph's identity is defined strictly by its topology (nodes and edges) rather than its spatial layout.

2.1 Task Taxonomy

The VGA consists of six distinct tasks involving graphs with 8–9 nodes, categorized by their logical complexity:

  • Isomorphism (Easy & Hard): Determining if two differently drawn graphs share the same topology.
  • Path Finding: Identifying Hamiltonian paths or calculating the shortest path between specific nodes.
  • Cycle Finding: Detecting Hamiltonian cycles or identifying the largest chordless cycle.

2.2 Experimental Methodology

The benchmark evaluates three classes of models:

  1. Supervised Vision Models: ViT, Swin Transformer, and ConvNeXt (fine-tuned on VGA tasks).
  2. Self-Supervised Models: DINOv2.
  3. Multimodal Large Language Models (MLLMs): GPT-4o, GPT-o1, and Claude 3.5 Sonnet (evaluated via zero-shot and Chain-of-Thought prompting).

To test generalization, researchers systematically varied graph layouts—for example, training models on a Kamada-Kawai layout and testing them on a Planar layout—to observe if the models learned the concept or merely the visual pattern.

3. Results and Analysis

3.1 The Conceptualization Gap

Human baselines achieved near-perfect accuracy (88–100%) across all tasks. In stark contrast, every AI model tested failed the graph isomorphism tasks, performing no better than random guessing. This suggests that even the most advanced models struggle to recognize when two different visual representations describe the same mathematical object.

3.2 Architectural Performance

Among supervised models, the convolutional architecture (ConvNeXt) consistently outperformed Transformer-based models (ViT, Swin) on non-failing tasks. For instance, on the Shortest Path task, ConvNeXt reached 82.4% accuracy on familiar layouts. However, performance dropped to 73.3% when the layout changed, indicating a lack of true representation invariance.

3.3 The Pseudo-Intelligence Trap in MLLMs

Analysis of advanced models like GPT-o1 revealed two critical anomalies that suggest a lack of logical reasoning:

  • The Middle-Score Anomaly: Models often produce mediocre scores on atomic tasks where a human would either succeed completely or fail entirely.
  • The Easier-Worse Anomaly: GPT-o1 performed worse on simpler versions of tasks (e.g., finding a path of length 1) than on complex versions (path of length 2). This indicates that the models are utilizing imitative-probabilistic reasoning—matching patterns seen in training data—rather than executing a logical algorithm.

4. Discussion and Future Work

The VGA benchmark demonstrates that current AI models lack the structural reasoning required for true visual conceptualization. While models can identify "leaf nodes" to rule out cycles, they cannot reliably trace paths or identify isomorphic structures.

Future research should focus on moving beyond probabilistic training toward methods like Reinforcement Learning, which may better mimic the biological learning processes required for structural understanding. Additionally, the VGA will be expanded to include other domains where structural invariants are vital, such as logic circuits and chemical structures, to further challenge the limits of multimodal reasoning.

5. Conclusion

The Visual Graph Arena highlights a massive disparity between human and AI visual reasoning. Until models can bridge the conceptualization gap and achieve representation invariance, their utility in complex, structure-dependent fields will remain limited. The VGA serves as a necessary benchmark for tracking progress toward more robust, conceptual AI.

Summary updated

Abstract

Recent advancements in multimodal large language models have driven breakthroughs in visual question answering. Yet, a critical gap persists, `conceptualization'—the ability to recognize and reason about the same concept despite variations in visual form, a basic ability of human reasoning. To address this challenge, we introduce the Visual Graph Arena (VGA), a dataset featuring six graph-based tasks designed to evaluate and improve AI systems’ capacity for visual abstraction. VGA uses diverse graph layouts (e.g., Kamada-Kawai vs. planar) to test reasoning independent of visual form. Experiments with state-of-the-art vision models and multimodal LLMs reveal a striking divide: humans achieved near-perfect accuracy across tasks, while models totally failed on isomorphism detection and showed limited success in path/cycle tasks. We further identify behavioral anomalies suggesting pseudo-intelligent pattern matching rather than genuine understanding. These findings underscore fundamental limitations in current AI models for visual understanding. By isolating the challenge of representation-invariant reasoning, the VGA provides a framework to drive progress toward human-like conceptualization in AI visual models. The Visual Graph Arena is available at: \href{https://vga.csail.mit.edu/}{vga.csail.mit.edu}.
BibTeX
@inproceedings{babaiee2025visual,
  author = {Zahra Babaiee and Peyman Kiasari and Daniela Rus and Radu Grosu},
  title = {{Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=BCJPAmlfxv},
}
Figure