Abstract
We introduce Mathador-LM, a new benchmark for evaluating the mathematical reasoning on large language models (LLMs), combining ruleset interpretation, planning, and problem-solving. This benchmark is inspired by the Mathador game, where the objective is to reach a target number using basic arithmetic operations on a given set of base numbers, following a simple set of rules. We show that, across leading LLMs, we obtain stable average performance while generating benchmark instances dynamically, following a target difficulty level. Thus, our benchmark alleviates concerns about test-set leakage into training data, an issue that often undermines popular benchmarks. Additionally, we conduct a comprehensive evaluation of both open and closed-source state-of-the-art LLMs on Mathador-LM. Our findings reveal that contemporary models struggle with Mathador-LM, scoring significantly lower than average 3rd graders. This stands in stark contrast to their strong performance on popular mathematical reasoning benchmarks.
Introduction: The AI Benchmark Crisis
The artificial intelligence community is currently facing a "benchmark crisis." Popular datasets such as GSM8K (Grade-School Math) and MATH are reaching saturation, with state-of-the-art models frequently scoring in the 90th percentile. However, it remains unclear whether these models are demonstrating genuine reasoning or simply retrieving patterns seen during training—a phenomenon known as test-set leakage or contamination.
Existing benchmarks often focus on word problems that can be solved through linguistic pattern matching. There is a critical need for benchmarks that require ruleset interpretation, multi-step planning, and combinatorial search. To address this, we introduce Mathador-LM, a rigorous evaluation framework adapted from the popular French mathematical game "Mathador."
The Mathador-LM Framework
The Mathador-LM benchmark challenges models to reach a target number using five base numbers and the four basic arithmetic operations (+, -, ×, ÷).
Methodology and Scoring
Unlike static benchmarks, Mathador-LM instances are generated dynamically. This ensures that the specific combination of numbers is likely unique to each evaluation run, making the benchmark "leak-proof." Scoring is based on the complexity of the operations used:
- Addition (+) and Multiplication (×): 1 point each.
- Subtraction (-): 2 points.
- Division (÷): 3 points.
- Mathador Bonus: 6 additional points are awarded for a "perfect" solution that uses all five numbers and all four operations exactly once.
Models are evaluated in a few-shot setting (typically 2 to 20 examples) to ensure they understand the rules and the required output format. We also introduce a formal difficulty measure based on the density of high-scoring solutions, allowing for the creation of "easy," "medium," and "hard" test sets.
Results and Analysis
We compared LLM performance on Mathador-LM against industry standards like MMLU and GSM8K across a wide range of models, including the Llama-3 family, Qwen2, Phi-3, GPT-4, and the Claude-3 series.
The "Reasoning Gap"
The most striking finding is the significant performance gap between AI and humans. While state-of-the-art LLMs score below 15% on average, human 3rd graders average 43.7% on the same task. This suggests that LLM mathematical "intelligence" is remarkably brittle when stripped of familiar word-problem templates.
Key Findings
- Scaling Laws: While larger models (70B+) outperform smaller ones (<3B), the improvement is marginal compared to the gap with human performance. For instance, GPT-4 and Claude-3 Haiku both scored below 7%.
- Constraint Failures: Error analysis reveals that over 60% of failures are due to "illegal operand" errors—using numbers not provided or reusing numbers already exhausted—rather than incorrect arithmetic.
- Stability: Despite the dynamic nature of the benchmark, model performance remains stable across different randomly generated sets of the same difficulty, confirming the reliability of dynamic evaluation.
Conclusion and Future Work
Mathador-LM highlights a significant deficiency in the planning and constraint-satisfaction capabilities of current LLMs. While these models excel at pattern matching in standard benchmarks, they struggle with search-based arithmetic tasks. Future research will explore whether advanced prompting techniques like Chain-of-Thought (CoT) or targeted fine-tuning can bridge this reasoning gap.
Summary updated