Introduction
While Large Language Models (LLMs) have demonstrated impressive capabilities in mathematical reasoning, current evaluation methods often fail to distinguish between genuine logical derivation and heuristic-driven guessing. Existing benchmarks frequently focus on a narrow set of popular topics or basic word problems, creating significant blind spots in our understanding of LLM intelligence. To address these issues, researchers have introduced the Mathematical Topics Tree (MaTT), a rigorous new benchmark designed to evaluate both the breadth and depth of mathematical reasoning across the undergraduate and graduate landscape.
The Mathematical Topics Tree (MaTT) Benchmark
The MaTT benchmark is structured as a hierarchical tree, encompassing 12 major branches of mathematics divided equally between Pure and Applied fields. The dataset consists of 1,958 manually curated questions derived from high-level reference books to ensure difficulty and minimize the likelihood of data contamination from common training sets.
The benchmark's architecture allows for a granular analysis of model performance across 772 nodes and 625 leaf nodes. Topics range from Abstract Algebra and Number Theory to Operations Research and Information Theory.
Evaluation Methodology
The researchers evaluated several prominent models, including GPT-4, ChatGPT, Mistral, Llama 3.1, and OpenAI's o1-mini. The evaluation protocol employed three distinct formats:
- Multiple-choice with Chain-of-Thought (CoT).
- Multiple-choice without CoT.
- Open-ended (no choices provided).
A critical component of the study was a manual reasoning audit. The authors reviewed the step-by-step explanations generated by GPT-4 to determine if the underlying logic was sound, even when the final answer was correct.
Key Findings and Results
The results of the MaTT benchmark reveal a significant gap between perceived and actual reasoning capabilities.
The "Choice" Crutch and Choice Engineering
A primary insight is the heavy dependence of models on multiple-choice options. When these options were removed, performance plummeted across the board. For instance, GPT-4’s accuracy dropped by nearly 16 percentage points, while Mistral’s performance fell from 23.1% to just 7.0%. This suggests that models often engage in "choice engineering"—using shortcuts like plugging in provided answers or picking extreme values in optimization problems—rather than solving from first principles.
The Reasoning Illusion
The manual audit of GPT-4's correct answers found that only 53.3% were supported by complete and accurate reasoning. The remaining "correct" answers relied on weak or circular logic. Furthermore, the addition of Chain-of-Thought (CoT) prompting provided negligible improvements, suggesting that CoT cannot compensate for a lack of fundamental conceptual understanding.
Comparative Model Performance
As shown in the table below, OpenAI's o1-mini currently leads the benchmark with an overall accuracy of 79.2%, representing a substantial leap over GPT-4's 54.0%.
Performance varied significantly by subject. While models generally performed better in Algebra and Calculus, they struggled in more abstract or "creative" fields such as Combinatorics, Logic, and Game Theory.
Conclusion
The MaTT benchmark exposes the limitations of current LLMs, highlighting a tendency toward memorization and heuristic guessing over genuine mathematical thinking. The significant performance drop in open-ended formats suggests that current leaderboards may overestimate model intelligence. Future research and benchmarking should move away from multiple-choice formats and focus on evaluating the integrity of the reasoning process itself.
Summary updated