VisNumBench: Evaluating Number Sense of Multimodal Large Language Models

Tengjin Weng, Jingyi Wang, Wenhao Jiang, Zhong Ming

While Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in complex symbolic reasoning—such as solving calculus problems or interpreting intricate charts—they often struggle with the foundational cognitive ability known as "number sense." This intuitive capacity allows humans to estimate quantities, angles, and sizes from visual cues without explicit counting or measurement. This paper introduces VisNumBench, a comprehensive benchmark designed to evaluate this perceptual-numerical gap in state-of-the-art models.

The Challenge of Intuitive Estimation

Humans possess an innate number sense that enables us to estimate the number of items in a container or the steepness of a slope at a glance. Current MLLM benchmarks primarily focus on structured mathematical reasoning, overlooking this grounded understanding of the physical world. The authors argue that achieving human-like intelligence requires MLLMs to move beyond symbolic manipulation toward a more intuitive, perceptual understanding of numerical relationships.

Figure 1: The VisNumBench framework evaluates number sense through observation, intuitive reasoning, and synthesis across various visual attributes.

Methodology and Dataset

The researchers developed the Visual Number Benchmark (VisNumBench), comprising approximately 1,900 tasks across seven visual numerical attributes: angle, scale, length, quantity, depth, area, and volume. The evaluation utilizes four distinct task types to probe different aspects of numerical intuition:

  1. Value Estimation: Providing a specific numerical value.
  2. Range Estimation: Identifying the correct numerical interval.
  3. Value Comparison: Determining relative size or quantity between objects.
  4. Multiplicative Estimation: Estimating ratios between visual elements.

To ensure robustness, the benchmark is divided into two subsets: VisNumBench-Synthetic, featuring mathematically defined relationships generated via scripts, and VisNumBench-Real, consisting of real-world images with natural lighting, occlusions, and complex backgrounds.

Key Results and the "Intuition Gap"

The evaluation of 17 prominent MLLMs, including GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL, revealed a significant performance disparity between machines and humans. While humans achieved an near-perfect average score of 96.3%, the highest-performing model, Gemini 2.0 Flash, reached only 57.1%. Among open-source models, Qwen2.5-VL-72B led the field at 56.0%.

Figure 6: Radar charts illustrating the performance gap between humans and various MLLMs across synthetic and real-world benchmarks.

Critical Insights

  • Math Specialization is Insufficient: Models fine-tuned on extensive mathematical datasets showed negligible improvement in number sense. This suggests that intuitive estimation is a distinct perceptual skill rather than a byproduct of logical training.
  • The Failure of Chain-of-Thought (CoT): Contrary to its typical benefits in LLMs, CoT prompting often decreased performance in VisNumBench. Models frequently "hallucinated" incorrect intermediate perceptual steps, leading to erroneous final estimates.
  • Attribute Difficulty: Angle perception emerged as the most challenging task for all models, with accuracy often approaching random chance. Quantity estimation was relatively easier, likely due to the abundance of counting-related data in standard training sets.
  • Nature of Errors: An error analysis of Gemini 2.0 Flash indicated that over 70% of its mistakes were "numerical intuition errors"—the model correctly identified the objects but failed to accurately estimate their spatial or numerical relationships.

Conclusion and Future Directions

The findings from VisNumBench highlight a fundamental limitation in current vision-language architectures. Existing vision encoders may not be optimized for high-precision spatial and numerical grounding. Future research should focus on developing specialized vision encoders that prioritize these relationships, creating larger "perceptual-numerical" datasets for pre-training, and exploring reinforcement learning techniques specifically tailored for intuitive estimation tasks.

Summary updated

BibTeX
@inproceedings{weng2025visnumbench,
  author = {Tengjin Weng and Jingyi Wang and Wenhao Jiang and Zhong Ming},
  title = {{VisNumBench: Evaluating Number Sense of Multimodal Large Language Models}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure