TL;DR
This paper introduces HighMATH, a new, more challenging benchmark for evaluating the math reasoning capabilities of large language models (LLMs). HighMATH reveals that current LLMs, despite achieving high scores on existing benchmarks, struggle significantly with problems requiring deeper understanding and the integration of multiple mathematical concepts, showing accuracy drops of over 30%.
What Problem Does This Paper Address?
The field of large language models (LLMs) has seen remarkable progress in mathematical reasoning, with many models now achieving over 90% accuracy on benchmarks like MATH. This rapid advancement raises a critical question: are these benchmarks still sufficient to truly evaluate and differentiate the capabilities of the latest LLMs? The authors argue that existing benchmarks, such as GSM8K and MATH, are becoming saturated. They point out that many of these datasets originate from elementary or middle school math, or focus on problems with limited knowledge points, failing to capture the complexity and breadth of reasoning required for advanced mathematical problem-solving. This gap necessitates a more challenging evaluation to push the boundaries of LLM capabilities.
Methods
The core contribution of this paper is the creation of the HighMATH dataset. This benchmark was curated with three key criteria:
- Problem Complexity: Problems are designed to be more intricate and require multiple reasoning steps.
- Knowledge Domain Diversity: The dataset covers a broad spectrum of high school mathematics, encompassing 8 subjects and 7 difficulty levels.
- Fine-grained Annotations: Each problem is annotated with detailed information, including subject, difficulty, and the specific knowledge points involved.
To create HighMATH, the authors collected 5,293 problems from Chinese senior high school mathematics exams published in 2024. They processed these using OCR and manual annotation, ensuring accuracy and consistency. They also created a subset called HighMATH-HARD for problems with multiple sub-questions, further increasing the challenge.
The evaluation methodology involved testing 20 LLMs across four settings: zero-shot, majority vote, Pass@8, and with a Python executor. This multi-faceted approach aims to provide a comprehensive understanding of model performance.
Datasets
- HighMATH: A newly introduced benchmark comprising 5,293 problems from Chinese senior high school mathematics exams. It covers 8 subjects and 7 difficulty levels, with an average of 2.4 knowledge points per problem.
- HighMATH-HARD: A subset of HighMATH containing problems with multiple interconnected sub-questions.
- MATH: Used as a baseline for comparison. The paper notes that HighMATH contains a higher average number of knowledge points per problem and subject compared to MATH.
Key Insights and Ideas
- Existing Benchmarks are Becoming Obsolete: The most striking finding is the significant drop in accuracy (over 30%) for advanced LLMs on HighMATH compared to benchmarks like MATH. This strongly suggests that current LLMs have reached a plateau on older datasets and HighMATH is a more effective discriminator.
- Breadth and Depth Matter: HighMATH's design, emphasizing both the breadth of mathematical topics and the depth of knowledge points within each problem, is crucial for revealing LLM limitations. Problems requiring the integration of multiple concepts are particularly challenging.
- Smaller LLMs Show Promise: The study indicates that smaller, properly trained LLMs can exhibit significant mathematical reasoning potential. For instance, the Pass@8 accuracy of a 1.5B model surpassed that of many 7B models, suggesting that focused training can be highly effective.
- Multi-step Reasoning is a Bottleneck: The accuracy drop is more pronounced for multi-question problems, highlighting LLMs' difficulties in maintaining reasoning chains and dependencies across multiple steps.
- Language is Not the Primary Barrier: While accuracy slightly decreased on translated problems, the impact was not significant, suggesting that HighMATH's challenges are inherent to mathematical reasoning rather than specific to the Chinese language.
Key Results
- Advanced LLMs, including "o1-like" models, experienced accuracy drops of over 30% on HighMATH compared to previous benchmarks.
- The best-performing model on HighMATH achieved an accuracy of 62% (under Pass@8 evaluation).
- Models performed consistently across different subjects, but accuracy decreased significantly with increasing difficulty levels.
- HighMATH problems have an average of 2.4 knowledge points, compared to 1.59 for MATH.
Limitations and Future Work
- Scope: HighMATH currently focuses on high school mathematics and does not cover university-level math or Olympiad problems.
- Step-by-Step Evaluation: The authors suggest that future work should focus on evaluating each reasoning step generated by the model, not just the final answer, to better understand complex logical reasoning processes.
- Data Contamination: While efforts were made to mitigate data contamination (e.g., using scanned PDFs), the authors acknowledge that static benchmarks are always prone to this issue.
- Model Size vs. Performance: While larger models generally perform better on harder problems, the study observed that simply increasing model size doesn't always lead to proportional improvements, especially for multi-question problems. This suggests that architectural and training strategies are as important as scale.
Summary updated