MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, Jianfeng Gao

TL;DR

This paper introduces MathVista, a new benchmark designed to rigorously evaluate the mathematical reasoning capabilities of foundation models (like LLMs and LMMs) when faced with visual contexts. It highlights that even advanced models like GPT-4V still significantly lag behind human performance, underscoring the need for further research in this critical area for developing more capable AI assistants.

What Problem Does This Paper Address?

While Large Language Models (LLMs) and Large Multimodal Models (LMMs) have shown impressive progress in various tasks, their ability to perform mathematical reasoning in visual contexts remains largely unexamined. This is a crucial gap because many real-world applications, from education and data analysis to scientific discovery, require models to understand and reason about mathematical concepts presented visually (e.g., in charts, diagrams, or images). Existing benchmarks often focus on text-only math problems or simple visual question answering, leaving a need for a comprehensive evaluation that combines both visual understanding and complex mathematical reasoning.

Methods

The authors created MathVista, a benchmark that consolidates 6,141 examples from 28 existing datasets and introduces 3 new ones: IQTest (for logical reasoning on puzzle figures), FunctionQA (for algebraic reasoning on function plots), and PaperQA (for scientific reasoning on academic figures). They established a taxonomy of seven mathematical reasoning types (algebraic, arithmetic, geometry, logical, numeric common sense, scientific, statistical) and five primary task types (figure question answering, geometry problem solving, math word problem, textbook question answering, visual question answering). The benchmark is designed to require fine-grained visual understanding and compositional reasoning, going beyond simple OCR or image captioning.

Datasets

  • Existing Datasets: 28 datasets were curated, including 9 math-targeted question answering datasets and 19 general visual question answering (VQA) datasets that contain math-related instances.
  • Newly Introduced Datasets:
    • IQTest: Focuses on logical reasoning with puzzle test figures.
  • FunctionQA: Evaluates algebraic reasoning on functional plots.
  • PaperQA: Assesses scientific reasoning using figures from academic papers.

Key Insights and Ideas

  • Visual Math is Hard: Even state-of-the-art models like GPT-4V, which excel in many areas, still fall significantly short of human performance on MathVista, highlighting the difficulty of integrating visual understanding with complex mathematical reasoning.
  • GPT-4V's Strengths and Weaknesses: While GPT-4V shows remarkable progress, it struggles with certain visual contexts (e.g., low-resolution figures, multiple functions) and specific reasoning types (e.g., logical reasoning, numeric common sense).
  • The Importance of Compositional Reasoning: Many MathVista problems require combining multiple pieces of visual information and applying several reasoning steps, a capability that current models are still developing.
  • Self-Verification Potential: GPT-4V exhibits an emergent ability for self-verification, where it can inspect its own reasoning and attempt to correct mistakes, a promising avenue for future research.

Key Results

  • GPT-4V leads, but with a gap: GPT-4V achieved the highest accuracy at 49.9%, significantly outperforming Multimodal Bard (34.8%). However, this is still 10.4% lower than human performance (60.3%).
  • LLMs struggle without visual context: Text-only LLMs performed poorly, underscoring the need for multimodal capabilities.
  • Augmented LLMs improve: LLMs augmented with image captions and OCR text showed improved performance but still lagged behind dedicated LMMs.
  • Open-source LMMs lag: Models like LLaVA and IDEFICS showed underwhelming performance, indicating a need for better architectures and training strategies for math-focused multimodal tasks.

Limitations and Future Work

  • Dataset Coverage: While MathVista is comprehensive, there might be gaps in representing certain mathematical problems and visual contexts. The scalability to other domains is also a concern.
  • Annotation Heterogeneity: Annotations were sourced from various providers, leading to a lack of a unified format.
  • Focus on Recent Models: The study focused on the most recent models, and future work could incorporate a broader range of models.

The authors suggest future work should focus on developing models that can seamlessly integrate mathematical reasoning with visual comprehension, exploring GPT-4V's self-verification and self-consistency abilities, and creating more robust, unified annotations for benchmarks.

Summary updated

Abstract

Large Language Models (LLMs) and Large Multimodal Models (LMMs) exhibit impressive problem-solving skills in many tasks and domains, but their ability in mathematical reasoning in visual contexts has not been systematically studied. To bridge this gap, we present MathVista, a benchmark designed to combine challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets (i.e., IQTest, FunctionQA, and PaperQA). Completing these tasks requires fine-grained, deep visual understanding and compositional reasoning, which all state-of-the-art foundation models find challenging. With MathVista, we have conducted a comprehensive, quantitative evaluation of 12 prominent foundation models. The best-performing GPT-4V model achieves an overall accuracy of 49.9%, substantially outperforming Bard, the second-best performer, by 15.1%. Our in-depth analysis reveals that the superiority of GPT-4V is mainly attributed to its enhanced visual perception and mathematical reasoning. However, GPT-4V still falls short of human performance by 10.4%, as it often struggles to understand complex figures and perform rigorous reasoning. This significant gap underscores the critical role that MathVista will play in the development of general-purpose AI agents capable of tackling mathematically intensive and visually rich real-world tasks. We further explore the new ability of self-verification, the application of self-consistency, and the interactive chatbot capabilities of GPT-4V, highlighting its promising potential for future research. The project is available at https://mathvista.github.io/.
BibTeX
@inproceedings{lu2024mathvista,
  author = {Pan Lu and Hritik Bansal and Tony Xia and Jiacheng Liu and Chunyuan Li and Hannaneh Hajishirzi and Hao Cheng and Kai-Wei Chang and Michel Galley and Jianfeng Gao},
  title = {{MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=KUNzEQMWU7},
}
Figure