Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models

Eldar Kurtic, Amir Moeini, Dan Alistarh

Abstract

We introduce Mathador-LM, a new benchmark for evaluating the mathematical reasoning on large language models (LLMs), combining ruleset interpretation, planning, and problem-solving. This benchmark is inspired by the Mathador game, where the objective is to reach a target number using basic arithmetic operations on a given set of base numbers, following a simple set of rules. We show that, across leading LLMs, we obtain stable average performance while generating benchmark instances dynamically, following a target difficulty level. Thus, our benchmark alleviates concerns about test-set leakage into training data, an issue that often undermines popular benchmarks. Additionally, we conduct a comprehensive evaluation of both open and closed-source state-of-the-art LLMs on Mathador-LM. Our findings reveal that contemporary models struggle with Mathador-LM, scoring significantly lower than average 3rd graders. This stands in stark contrast to their strong performance on popular mathematical reasoning benchmarks.

Introduction: The Saturation Crisis in AI Evaluation

The AI community is currently facing a "saturation" crisis in mathematical evaluation. Popular benchmarks like GSM8K and MMLU are seeing near-perfect scores from state-of-the-art models. However, these static datasets suffer from two major flaws: test-set leakage and a reliance on pattern matching over true reasoning. Because these datasets are public and static, they often end up in the training data of new models, leading to inflated performance that reflects memorization. Furthermore, high scores on word problems don't necessarily mean a model can "think" through novel constraints-based problems.

Mathador-LM addresses these issues by introducing a dynamic benchmark—one where problems are generated on-the-fly—ensuring the model is always facing a "fresh" challenge that it couldn't have seen during training.

The Mathador Game and Methodology

The benchmark is based on the Mathador game. In this game, a player is given five base numbers and a target number. The goal is to reach the target using the four basic arithmetic operations (+, -, ×, ÷).

Example of a simple vs. high-scoring 'Mathador' solution.

Key procedural rules include: * Each base number can be used only once. * Intermediate results must be non-negative integers. * Scoring: Points are awarded for reaching the target, with bonuses for using "harder" operators like division and a massive "Mathador Bonus" for a perfect solution that uses all five numbers and all four operators exactly once.

The researchers formalized a Difficulty Measure for these problems based on the number and quality of possible solutions. This allows them to generate "easy," "medium," and "hard" sets dynamically while maintaining stable evaluation metrics.

Key Results and Insights

The most striking insight from our evaluation is the performance gap between humans and AI. Human 3rd graders (8-year-olds) in France average a score of 43.7% on this game. In contrast, even the most advanced LLMs struggle significantly.

Comparison of LLM performance on Mathador-LM vs. standard benchmarks MMLU and GSM8K.

Failure to Follow Rules

Models don't just fail at the math; they fail at the logic of the game. Over 60% of errors were "Illegal Operand" errors—meaning the model used a number it wasn't given or reused a number it had already spent.

Scaling and Performance

  • Top Tier: Llama-3-70B, Qwen2-72B, and Claude-3-Opus lead the field but only achieve 10-15% accuracy.
  • Mid Tier: Models in the 7-8B range (like Llama-3-8B or Qwen2-7B) score between 5-7%.
  • Surprising Underperformers: GPT-4 and Claude-3-Haiku both scored below 7%, performing on par with much smaller open-source models.
  • Small Models: Models below 3B parameters (like Phi-3-mini) obtain negligible accuracy.

While larger models perform better, the returns are diminishing. Moving from 8B to 70B parameters provides a boost, but even the largest models remain far from human-level proficiency in this specific type of combinatorial planning.

Conclusion and Future Work

Mathador-LM proves that a dynamic benchmark can be just as reliable as a static one. Model performance remained stable across different randomly generated sets of the same difficulty level, suggesting that "one-time" benchmarks are a viable path forward for AI evaluation.

While Mathador-LM focuses on search-based arithmetic, future work will explore advanced prompting techniques (like Tree-of-Thought), fine-tuning for combinatorial search, and increasing problem complexity to further challenge future models.

Summary updated

Abstract

We introduce Mathador-LM, a new benchmark for evaluating the mathematical reasoning on large language models (LLMs), combining ruleset interpretation, planning, and problem-solving. This benchmark is inspired by the Mathador game, where the objective is to reach a target number using basic arithmetic operations on a given set of base numbers, following a simple set of rules. We show that, across leading LLMs, we obtain stable average performance while generating benchmark instances \emph{dynamically}, following a target difficulty level. Thus, our benchmark alleviates concerns about test-set leakage into training data, an issue that often undermines popular benchmarks. Additionally, we conduct a comprehensive evaluation of both open and closed-source state-of-the-art LLMs on Mathador-LM. Our findings reveal that contemporary models struggle with Mathador-LM, scoring significantly lower than average 3rd graders. This stands in stark contrast to their strong performance on popular mathematical reasoning benchmarks. The implementation of Mathador-LM benchmark is available at \href{https://github.com/IST-DASLab/Mathador-LM}{github.com/IST-DASLab/Mathador-LM}.
BibTeX
@inproceedings{kurtic2024mathador,
  author = {Eldar Kurtic and Amir Moeini and Dan Alistarh},
  title = {{Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=KpehAKtitF},
}
Figure