UTMath: A Benchmark for Math Evaluation with Unit Test

Bo Yang, Qingping Yang, Yingwei Ma, Runtao LiuEMNLP 2025

TL;DR

This paper introduces UTMath, a novel benchmark for evaluating mathematical reasoning in LLMs that uses unit tests to assess both accuracy and generalization across varied problem instances. They also propose RCoT, a prompting strategy that decouples reasoning from code generation, leading to more efficient and robust solutions.

What Problem Does This Paper Address?

Evaluating the mathematical reasoning capabilities of Large Language Models (LLMs) is crucial for advancing towards Artificial General Intelligence (AGI). However, existing benchmarks often rely on single-instance problems with fixed numbers, allowing LLMs to succeed through pattern matching or memorization rather than genuine deductive reasoning. This limits their ability to assess how well models generalize to structurally similar but numerically different problems. Furthermore, many evaluation protocols are susceptible to inconsistencies due to the stochastic nature of LLM outputs.

Methods

The authors introduce UTMath, a benchmark built upon the On-Line Encyclopedia of Integer Sequences (OEIS). It comprises 1,053 problems across 9 mathematical domains, each with an average of 68 varied test cases. This unit-test-driven framework requires LLMs to generate generalizable code solutions that pass multiple test cases, moving beyond simple answer matching.

They also propose Reasoning-to-Code Thoughts (RCoT), a two-turn prompting strategy. In the first turn, the LLM focuses solely on mathematical reasoning without generating code. In the second turn, it generates executable code based on the reasoning from the first turn. This decoupling aims to encourage deeper reasoning and produce more efficient solutions.

Datasets

  • UTMath Benchmark: A newly introduced benchmark consisting of 1,053 problems derived from OEIS sequences. Each problem has an average of 68 test cases.
  • UTMath-Train: A companion training set with 70,000 samples, generated under the same protocol as UTMath.
  • OEIS (On-Line Encyclopedia of Integer Sequences): The source for the UTMath benchmark problems.

Key Insights and Ideas

  • The limitations of single-instance evaluation: LLMs can easily overfit to specific problem instances, masking a lack of true reasoning ability. UTMath's multi-test case approach directly addresses this.
  • The power of decoupling reasoning and coding: RCoT demonstrates that separating these two tasks allows LLMs to perform more thorough reasoning, leading to better solutions and improved efficiency.
  • The importance of "hard cases": Mining later terms of OEIS sequences as "hard test cases" effectively filters out simplistic or brute-force solutions, revealing deeper reasoning capabilities.
  • Domain-specific challenges: LLMs still struggle significantly with certain mathematical domains like Graph Theory and Geometry, indicating areas for future research.

Key Results

  • UTMath is challenging: The best-performing model, o1-mini, only achieved 32.57% accuracy on UTMath, highlighting the benchmark's difficulty. GPT-40 scored 26.93%.
  • RCoT outperforms PoT: RCoT achieved higher pass@5 performance across 8 LLMs and led to more efficient solutions, with some models showing over 36% efficiency improvement.
  • Hard cases significantly reduce performance: Model performance drops notably on hard test cases, demonstrating their effectiveness in evaluating robust reasoning.
  • Domain-specific performance variations: LLMs performed poorly in Graph Theory and Geometry, with accuracy rates below 9%.

Limitations and Future Work

  • Runtime dependency: The evaluation metrics are influenced by the runtime of the generated code, making absolute numerical results incomparable across different machines. The authors recommend using the same machine for testing and replication.
  • Limited package support: Only a set of common packages (like sympy) were installed in the testing environment to prevent harmful code execution.
  • Focus on reasoning quality: While RCoT demonstrates the critical role of reasoning quality, the authors did not further explore methods for enhancing it, suggesting this as a future research direction.
  • Benchmark inclusivity: UTMath may not fully represent diverse cultural or educational perspectives, and the authors encourage the development of benchmarks that incorporate a broader array of reasoning styles.

Summary updated

BibTeX
@inproceedings{yang2025utmath,
  author = {Bo Yang and Qingping Yang and Yingwei Ma and Runtao Liu},
  title = {{UTMath: A Benchmark for Math Evaluation with Unit Test}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure