MathVista: Evaluating Math Reasoning in Visual Contexts with GPT-4V, Bard, and Other Large Multimodal Models

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, Jianfeng Gao

Abstract

Large Language Models (LLMs) and Large Multimodal Models (LMMs) have demonstrated impressive problem-solving skills, yet their ability to perform mathematical reasoning within visual contexts has lacked systematic study. We present MathVista, a benchmark of 6,141 examples derived from 28 existing datasets and 3 newly created ones (IQTest, FunctionQA, and PaperQA). Our comprehensive evaluation of 12 foundation models reveals that while GPT-4V leads with 49.9% accuracy—outperforming Bard by 15.1%—it still falls 10.4% short of human performance. This gap highlights the ongoing challenge of combining fine-grained visual perception with rigorous compositional reasoning.

Introduction: Bridging the Vision-Logic Gap

While LLMs like ChatGPT excel at text-based logic, they are often "blind" to the visual world. Conversely, standard Visual Question Answering (VQA) datasets typically focus on simple object recognition rather than complex mathematical logic. Real-world reasoning often requires interpreting function plots, reading statistical charts, and understanding geometric diagrams. MathVista was created to bridge this gap, challenging models to perform "compositional reasoning"—the ability to first perceive visual details and then use those details to execute multi-step mathematical operations.

The MathVista Benchmark

MathVista is a "mega-benchmark" designed to test a wide array of skills. The authors developed a taxonomy including 7 reasoning types (Algebraic, Arithmetic, Geometry, Logical, Numeric Common Sense, Scientific, and Statistical) and 5 primary tasks (Figure QA, Geometry Problem Solving, Math Word Problems, Textbook QA, and VQA).

To fill specific gaps in existing data, three new datasets were introduced:

  • IQTest: Logic puzzles requiring pattern prediction.
  • FunctionQA: Questions about algebraic properties of functional plots.
  • PaperQA: Scientific reasoning based on figures from academic papers.
Examples of the new datasets introduced in MathVista: IQTest, FunctionQA, and PaperQA.

Evaluation is performed via a three-stage pipeline: response generation, automated answer extraction using GPT-4, and score calculation against ground truth. This ensures that models are judged on their final answers rather than just keyword matching.

Key Results and Insights

The evaluation of 12 prominent foundation models reveals a significant performance hierarchy and several critical bottlenecks in current AI capabilities.

Radar charts illustrating model performance across various mathematical reasoning types and visual contexts.

GPT-4V: A New State-of-the-Art

GPT-4V achieved an overall accuracy of 49.9%, establishing a substantial lead over the second-best performer, Multimodal Bard (34.8%). A notable capability of GPT-4V is its "self-verification," where it occasionally identifies and corrects its own logical errors mid-response. Interestingly, GPT-4V surpassed human performance in specific areas like algebraic reasoning and function plot interpretation, though it lagged in "common sense" visual tasks.

The Perception-Reasoning Bottleneck

A primary insight from MathVista is that many models fail not due to a lack of mathematical knowledge, but due to poor visual perception. For instance, a model might correctly identify a formula but fail because it misreads the numerical values on a chart's axis. This "perception gap" is particularly pronounced in open-source models like LLaVA, which performed only slightly better than random chance (17.9%).

The Human Baseline

Despite the advancements of GPT-4V, the human baseline remains higher at 60.3%. Models still struggle with hallucinations—inventing visual facts—and complex spatial tasks, such as counting small objects in crowded scenes or solving intricate geometric puzzles. Furthermore, providing text-only models with perfect image captions (33.9%) resulted in lower performance than native multimodal models, confirming that direct visual perception is vital for these tasks.

Conclusion

MathVista underscores the critical role of visual context in mathematical reasoning. While GPT-4V represents a significant leap forward, the 10.4% gap compared to human performance indicates that general-purpose AI agents still have much to learn. By releasing MathVista as an open-source project, the authors aim to catalyze the development of models that can truly "see" and reason through the complexities of the physical and mathematical world.

Summary updated

Abstract

Large Language Models (LLMs) and Large Multimodal Models (LMMs) exhibit impressive problem-solving skills in many tasks and domains, but their ability in mathematical reasoning in visual contexts has not been systematically studied. To bridge this gap, we present MathVista, a benchmark designed to combine challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets (i.e., IQTest, FunctionQA, and PaperQA). Completing these tasks requires fine-grained, deep visual understanding and compositional reasoning, which all state-of-the-art foundation models find challenging. With MathVista, we have conducted a comprehensive, quantitative evaluation of 12 prominent foundation models. The best-performing GPT-4V model achieves an overall accuracy of 49.9%, substantially outperforming Bard, the second-best performer, by 15.1%. Our in-depth analysis reveals that the superiority of GPT-4V is mainly attributed to its enhanced visual perception and mathematical reasoning. However, GPT-4V still falls short of human performance by 10.4%, as it often struggles to understand complex figures and perform rigorous reasoning. This significant gap underscores the critical role that MathVista will play in the development of general-purpose AI agents capable of tackling mathematically intensive and visually rich real-world tasks. We further explore the new ability of self-verification, the application of self-consistency, and the interactive chatbot capabilities of GPT-4V, highlighting its promising potential for future research. The project is available at https://mathvista.github.io/.
BibTeX
@inproceedings{lu2024mathvista,
  author = {Pan Lu and Hritik Bansal and Tony Xia and Jiacheng Liu and Chunyuan Li and Hannaneh Hajishirzi and Hao Cheng and Kai-Wei Chang and Michel Galley and Jianfeng Gao},
  title = {{MathVista: Evaluating Math Reasoning in Visual Contexts with GPT-4V, Bard, and Other Large Multimodal Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=KUNzEQMWU7},
}
Figure