U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in Large Language Models

Konstantin Chernyshev, Vitaliy Polshkov, Vlad Stepanov, Alex Myasnikov, Ekaterina Artemova, Alexei Miasnikov, Sergei TilgaACL 2025

TL;DR

This paper introduces U-MATH, a novel benchmark for evaluating the mathematical reasoning capabilities of Large Language Models (LLMs) at the university level, including a significant portion of visual problems. They also present μ-MATH, a meta-evaluation benchmark to assess the reliability of LLM judges, revealing current LLMs struggle with advanced math, especially visual reasoning, and that judging solutions is a distinct and challenging skill.

What Problem Does This Paper Address?

Evaluating the mathematical prowess of LLMs is crucial, but existing benchmarks often fall short. Many focus on school-level problems, are too small, or rely on multiple-choice formats, leading to rapid saturation and failing to capture the complexity of real-world mathematical tasks. Furthermore, the rise of multimodal LLMs necessitates benchmarks that include visual elements, and the evaluation of LLM-generated solutions is a significant challenge due to inherent biases and inconsistencies in LLM judges. This paper aims to fill these gaps by providing a more comprehensive and challenging university-level math benchmark and a tool to rigorously assess LLM evaluators.

Methods

The authors introduce two key contributions:

  • U-MATH Benchmark: This benchmark consists of 1,100 unpublished, open-ended university-level math problems sourced from actual US curricula. A significant portion (20%) includes visual elements like graphs and tables. The problems are designed to be calculation-intensive, focusing on reasoning rather than rote arithmetic.
  • μ-MATH Benchmark: This is a meta-evaluation benchmark derived from U-MATH. It comprises 1,084 tasks designed to assess the quality of LLM judges. It provides LLM-generated solutions (both correct and incorrect) for U-MATH problems, allowing for the evaluation of how well LLMs can judge the correctness of other LLMs' solutions.

For evaluation, they use Accuracy for U-MATH and Macro-averaged F1-score for μ-MATH, employing LLM judges for solution evaluation.

Datasets

  • U-MATH: A newly introduced benchmark of 1,100 university-level math problems, with 20% incorporating visual elements.
  • μ-MATH: A newly introduced meta-evaluation benchmark of 1,084 tasks derived from U-MATH, used to evaluate LLM judges.
  • Existing Benchmarks (for comparison): GSM8K, MATH, MMLU, MathQA, MathVista, MATH-V, We-Math, MathVerse, and others are mentioned as prior work.

Key Insights and Ideas

  • Visual Reasoning is a Major Hurdle: LLMs significantly underperform on visual math problems compared to text-only problems, even when the underlying concepts are similar. This highlights a critical area for improvement in multimodal LLMs.
  • Judging is a Distinct Skill: The ability to solve a math problem is not the same as the ability to judge the correctness of a solution. LLMs that are strong problem-solvers don't automatically become good judges, and vice-versa.
  • LLM Judges are Biased and Inconsistent: The μ-MATH benchmark reveals that LLM judges exhibit biases (e.g., favoring certain models' solutions) and can be inconsistent, even with advanced models. This necessitates careful meta-evaluation.
  • Specialization and Fine-tuning Matter: Models specifically fine-tuned for math (like Qwen2.5-Math) or those that have undergone continuous fine-tuning show improved performance, sometimes surpassing larger, general-purpose models.
  • CoT Prompting Impacts Judgment: While Chain-of-Thought (CoT) prompting generally improves judgment performance and reduces bias, its effectiveness varies across different model families.

Key Results

  • U-MATH Performance: Leading LLMs achieved a maximum accuracy of 93.1% on textual tasks but only 58.5% on visual tasks.
  • μ-MATH Performance: Even the most advanced models struggled with judgment, with the best achieving an imperfect F1-score of 90.1%.
  • Model Comparisons: Proprietary models generally outperformed open-weight models on U-MATH, especially in visual tasks, though they showed near-parity in text-only problems.
  • Bias in Judgment: Judges consistently showed better performance on solutions from certain models (e.g., Llama) compared to others (e.g., Qwen).
  • Trade-offs: A trade-off was observed between problem-solving and judgment capabilities, with some models excelling in one at the expense of the other.

Limitations and Future Work

  • Limited Scope of U-MATH: While covering six university subjects, U-MATH does not encompass the full range of advanced mathematical topics (e.g., complex analysis). The 20% visual problem proportion also limits the scope of visual reasoning evaluation.
  • Accuracy Metric Limitations: Accuracy discards valuable information and doesn't allow for finer-grained analysis.
  • LLM Judge Errors: Reliance on LLM judges introduces errors and biases that are quantified but not fully mitigated.
  • Future Work: The authors suggest developing assessment protocols for partial credit, bridging the gap between quantifying and controlling for auto-evaluation bias, and exploring adversarial task creation for meta-evaluations. They also emphasize the need for more advanced topics and a broader coverage of visual problems.

Summary updated

BibTeX
@inproceedings{chernyshev2025u,
  author = {Konstantin Chernyshev and Vitaliy Polshkov and Vlad Stepanov and Alex Myasnikov and Ekaterina Artemova and Alexei Miasnikov and Sergei Tilga},
  title = {{U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure