Abstract
Large Language Models (LLMs) and Large Multimodal Models (LMMs) have demonstrated impressive problem-solving skills, yet their ability to perform mathematical reasoning within visual contexts has lacked systematic study. We present MathVista, a benchmark of 6,141 examples derived from 28 existing datasets and 3 newly created ones (IQTest, FunctionQA, and PaperQA). Our comprehensive evaluation of 12 foundation models reveals that while GPT-4V leads with 49.9% accuracy—outperforming Bard by 15.1%—it still falls 10.4% short of human performance. This gap highlights the ongoing challenge of combining fine-grained visual perception with rigorous compositional reasoning.
Introduction: Bridging the Vision-Logic Gap
While LLMs like ChatGPT excel at text-based logic, they are often "blind" to the visual world. Conversely, standard Visual Question Answering (VQA) datasets typically focus on simple object recognition rather than complex mathematical logic. Real-world reasoning often requires interpreting function plots, reading statistical charts, and understanding geometric diagrams. MathVista was created to bridge this gap, challenging models to perform "compositional reasoning"—the ability to first perceive visual details and then use those details to execute multi-step mathematical operations.
The MathVista Benchmark
MathVista is a "mega-benchmark" designed to test a wide array of skills. The authors developed a taxonomy including 7 reasoning types (Algebraic, Arithmetic, Geometry, Logical, Numeric Common Sense, Scientific, and Statistical) and 5 primary tasks (Figure QA, Geometry Problem Solving, Math Word Problems, Textbook QA, and VQA).
To fill specific gaps in existing data, three new datasets were introduced:
- IQTest: Logic puzzles requiring pattern prediction.
- FunctionQA: Questions about algebraic properties of functional plots.
- PaperQA: Scientific reasoning based on figures from academic papers.
Evaluation is performed via a three-stage pipeline: response generation, automated answer extraction using GPT-4, and score calculation against ground truth. This ensures that models are judged on their final answers rather than just keyword matching.
Key Results and Insights
The evaluation of 12 prominent foundation models reveals a significant performance hierarchy and several critical bottlenecks in current AI capabilities.
GPT-4V: A New State-of-the-Art
GPT-4V achieved an overall accuracy of 49.9%, establishing a substantial lead over the second-best performer, Multimodal Bard (34.8%). A notable capability of GPT-4V is its "self-verification," where it occasionally identifies and corrects its own logical errors mid-response. Interestingly, GPT-4V surpassed human performance in specific areas like algebraic reasoning and function plot interpretation, though it lagged in "common sense" visual tasks.
The Perception-Reasoning Bottleneck
A primary insight from MathVista is that many models fail not due to a lack of mathematical knowledge, but due to poor visual perception. For instance, a model might correctly identify a formula but fail because it misreads the numerical values on a chart's axis. This "perception gap" is particularly pronounced in open-source models like LLaVA, which performed only slightly better than random chance (17.9%).
The Human Baseline
Despite the advancements of GPT-4V, the human baseline remains higher at 60.3%. Models still struggle with hallucinations—inventing visual facts—and complex spatial tasks, such as counting small objects in crowded scenes or solving intricate geometric puzzles. Furthermore, providing text-only models with perfect image captions (33.9%) resulted in lower performance than native multimodal models, confirming that direct visual perception is vital for these tasks.
Conclusion
MathVista underscores the critical role of visual context in mathematical reasoning. While GPT-4V represents a significant leap forward, the 10.4% gap compared to human performance indicates that general-purpose AI agents still have much to learn. By releasing MathVista as an open-source project, the authors aim to catalyze the development of models that can truly "see" and reason through the complexities of the physical and mathematical world.
Summary updated