Abstract
While Large Language Models (LLMs) have demonstrated proficiency in basic arithmetic, they often struggle with "knowledge-intensive" mathematics found in competitive exams and college-level courses. This paper introduces Theorem Rationale (TR), a training framework that teaches models to explicitly identify and apply mathematical theorems before solving complex problems. By fine-tuning models on a curated dataset of problem-theorem-solution triples, researchers significantly boosted the reasoning performance of open-source models like LLaMA-3, particularly in advanced mathematical domains.
The Knowledge Retrieval Problem
Current methods for improving LLM reasoning primarily rely on Chain-of-Thought (CoT), where a model "thinks out loud" to reach an answer. However, the authors argue that CoT lacks a crucial step used by humans: knowledge retrieval. When a human encounters a difficult geometry or calculus problem, they do not simply start calculating; they first identify which theorems (e.g., the Quadratic Formula or L'Hôpital's rule) are applicable. Without this explicit "theorem alignment," LLMs are prone to "hallucinating" intermediate steps or applying irrelevant mathematical rules, leading to logical failures.
Methodology: Theorem Rationale (TR)
The researchers developed a framework to transition from simple Question-Answer pairs to Theorem Rationale (TR) triples: (Question, Theorems, Solution).
1. TR Stimulation
Using GPT-4o as a teacher, the team transformed 37,000 standard math problems into TR triples. The model was prompted to first list the necessary theorems and then provide a solution strictly constrained by those theorems.
2. Theorem-oriented Instruction Evolution
To ensure the model truly internalizes mathematical concepts, the authors designed a hierarchical training strategy inspired by human pedagogy:
- Theorem Memorization (TM): The model is asked to explain a theorem given its name (e.g., "What is the Quadratic Formula?").
- Theorem Alignment (TA): A bidirectional task where the model either lists theorems for a given question or generates a sample question requiring a specific theorem.
- Theorem-based Problem Solving (TPS): The core task where the model must reflect on theorems as the first step of its reasoning path.
This structured approach ensures the reasoning path follows a logical sequence: $q_i \rightarrow t_i \rightarrow step_i^1 \rightarrow \dots \rightarrow step_i^L \rightarrow a_i$, where the final answer $a_i$ is grounded in the identified theorem $t_i$.
Experimental Setup and Datasets
The authors compiled a training set of 37,000 questions from diverse sources, including GSM8k (Primary/Middle School), Conic10k (High School), and OlympiadBench/TheoremQA (College/Olympiad). Evaluation was conducted on Out-of-Distribution (OOD) datasets to measure generalization, including SAT-Math, SciBench, JEEBench, and MMLU-pro-Math.
Key Insights and Results
The fine-tuned LLaMA-3-8B model (referred to as "Ours") showed consistent improvements across all benchmarks:
- Overall Performance: Achieved an average accuracy of 33.1%, outperforming the base LLaMA-3 (29.5%) and specialized models like MAmmoTH-CoT (22.2%).
- OOD Generalization: On the difficult SAT-Math dataset, the model improved from 54.1% to 61.8% (+7.7%).
- Advanced Math: On JEEBench (Indian Institute of Technology entrance exams), the model's accuracy more than doubled compared to previous open-source baselines.
- Data Efficiency: The study suggests that the quality and structure of the reasoning path are more important than volume; models trained on 30k TR samples often outperformed those trained on larger, unstructured CoT datasets.
Conclusion and Future Work
The study confirms that the "knowledge gap" in LLMs is often a failure of rule selection rather than calculation. Explicitly triggering theorem retrieval acts as a "logical anchor," making the reasoning process more transparent and interpretable. While the model still trails behind closed-source giants like GPT-4, the TR approach provides a scalable path toward autonomous, theorem-based learning. Future work will focus on expanding the theorem library and developing automatic theorem-based feedback systems.
Summary updated