TL;DR
This paper introduces VisNumBench, a novel benchmark designed to evaluate the "number sense" of multimodal large language models (MLLMs). Their findings reveal that even state-of-the-art MLLMs perform significantly below human levels in intuitive numerical estimation tasks, highlighting a critical gap in their understanding beyond explicit reasoning.
What Problem Does This Paper Address?
Humans possess an innate "number sense" that allows for intuitive perception, estimation, and manipulation of numerical information in everyday scenarios. This ability is crucial for grasping complex mathematical concepts and applying them practically. While MLLMs have made impressive strides in various multimodal tasks, existing benchmarks primarily focus on structured mathematical reasoning, abstract symbolic computation, or interpreting numerical data in textual contexts. They largely overlook this fundamental aspect of intuitive number sense, leaving a gap in our understanding of whether MLLMs truly comprehend numerical concepts or merely process them based on learned patterns. This paper aims to bridge that gap by evaluating MLLMs' intuitive number sense.
Methods
The core contribution is the introduction of VisNumBench, a comprehensive benchmark designed to evaluate the intuitive numerical abilities of MLLMs. VisNumBench is structured into two main components:
- VisNumBench-Synthetic: This subset uses controlled, synthetic images where numerical relationships are explicitly defined, allowing for precise control over data generation and minimizing biases.
- VisNumBench-Real: This subset comprises real-world images, offering a more complex and less controlled environment that better reflects practical scenarios.
VisNumBench covers seven fundamental visual numerical attributes: angle, length, scale, quantity, depth, area, and volume. It also incorporates four distinct modes of visual numerical estimation: value comparison, value estimation, range estimation, and multiplicative estimation. The dataset consists of approximately 1,900 multiple-choice question-answer pairs. The construction process involved generating figures using Python scripts, incorporating existing datasets, collecting data from web sources, and manually capturing real-world scenes. Distractors for multiple-choice questions were designed using both automated and manual methods to mimic human perceptual biases.
Datasets
The paper introduces VisNumBench, a newly created benchmark for evaluating MLLMs' number sense. It is divided into two subsets:
- VisNumBench-Synthetic: Generated using Python scripts for controlled environments.
- VisNumBench-Real: Collected from diverse sources including existing datasets, web collections, and manually captured images to represent real-world complexity.
The paper also references existing datasets used in the construction of VisNumBench, such as CLEVR-Math, MathVista, and TartanAir, among others.
Key Insights and Ideas
- MLLMs lack intuitive number sense: The most striking insight is that even the most advanced MLLMs, including proprietary models like GPT-4o and Gemini 2.0 Flash, perform significantly below human levels on intuitive numerical estimation tasks. This suggests a fundamental difference in how humans and current MLLMs process numerical information.
- Beyond explicit reasoning: The poor performance highlights that MLLMs often rely on explicit reasoning steps rather than perceptual intuition, which is a hallmark of human number sense. This raises questions about their genuine understanding of numerical concepts.
- Model size and general ability matter, but not enough: While larger models and those with broader general abilities show modest gains, these improvements are insufficient to bridge the gap with human performance.
- Multimodal mathematical and CoT models show limited gains: Adopting specialized multimodal mathematical models or multimodal Chain-of-Thought (CoT) approaches did not lead to substantial improvements in number sense abilities on this benchmark.
- Real-world complexity is a challenge: Performance on VisNumBench-Real is generally lower than on VisNumBench-Synthetic, indicating that the complexity and variability of real-world images pose a greater challenge for MLLMs.
- Quantity tasks are "easiest": Among the seven numerical attributes, quantity-related tasks appear to be the easiest for MLLMs, while angle-related tasks are the most difficult.
Key Results
- Human performance is near-perfect: Humans achieve near-perfect accuracy on VisNumBench.
- MLLM performance is poor: The best-performing MLLMs achieve average accuracies in the range of 40-60% on VisNumBench-Synthetic and 40-56% on VisNumBench-Real, significantly trailing human capabilities.
- Top performers: Among open-source models, Qwen2.5-VL-72B and InternVL2.5-78B show strong performance. For API-based models, Gemini 2.0 Flash and GPT-40 perform relatively well, though still far from human levels.
- Task-specific performance: Models show varying performance across different estimation tasks. For instance, LLaVA-v1.6-34B excels in multiplicative estimation, while Gemini 2.0 Flash and InternVL2.5-78B perform well in value comparison tasks.
Limitations and Future Work
- Limited number sense: The paper clearly demonstrates that current MLLMs have a significant deficiency in intuitive number sense.
- Need for specialized optimization: The authors suggest that to enhance number sense abilities within a short period, more specialized optimizations in data, training techniques, and model architecture are required.
- Historical model improvements: Experiments with historical models from the same family (e.g., Qwen-VL and InternVL) show that newer versions generally outperform older ones, indicating progress through improvements in data scale and quality, model architecture, and training strategies.
- Future directions: The paper implies a need for developing new training methodologies and architectures that specifically foster intuitive number sense, moving beyond purely explicit reasoning. Further research into how to imbue MLLMs with a more human-like perceptual understanding of numbers is crucial.
Summary updated