Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learning

An-Zi Yen, Wei-Ling HsuEMNLP 2023 · DOI

Abstract

As educational technology shifts toward personalized e-learning, the potential for Large Language Models (LLMs) to serve as autonomous mathematics tutors has gained significant attention. This paper investigates the mathematical and pedagogical capabilities of GPT-3.5-turbo using GRE-level word problems from the MathQA dataset. By employing a novel Teacher–Student framework, the researchers evaluate the model's ability to solve complex problems and provide corrective feedback. The findings reveal that while LLMs are proficient at generating fluent text, they struggle with high-level reasoning, 3D spatial logic, and accurate error identification, suggesting they are not yet ready for fully autonomous tutoring roles.

Introduction

The rapid advancement of LLMs has opened new frontiers in automated education. However, effective mathematics instruction requires more than just providing correct answers; it necessitates identifying a student's specific logical failures and offering tailored corrective guidance. This "pedagogical ability" remains a significant hurdle for current AI models. This study addresses three fundamental questions: how LLMs can assist in math learning, their proficiency in solving and explaining complex problems, and their capacity to act as teachers by rectifying student errors.

Methodology

The researchers utilized GPT-3.5-turbo to assess mathematical competence through three primary prompting strategies: Zero-shot, Few-shot, and Chain-of-Thought (CoT). To specifically measure pedagogical utility, they developed a Teacher–Student Framework.

The Teacher-Student framework used to evaluate the LLM's ability to identify and correct mathematical errors.

In this setup, one instance of the model acts as a "student" to generate an initial solution. A second instance, acting as the "teacher," reviews the student's work against the original problem. The teacher model is tasked with determining the correctness of the answer and explaining any detected errors. To validate these findings, the study included human evaluation by a mathematics expert and five college students, who provided human-authored answers for the model to critique.

Datasets and Complexity

The study leverages MathQA, a challenging dataset of GRE-level math word problems. Unlike common benchmarks like GSM8K, MathQA includes advanced topics such as geometry (including 3D spatial reasoning), physics-based problems, probability, and complex algebra. This complexity was chosen to push the boundaries of LLM reasoning beyond simple arithmetic.

Key Insights and Reasoning Failures

The analysis revealed several critical weaknesses in the LLM's mathematical reasoning:

  • Conceptual Misunderstanding: The most frequent error type (36.5%) was "Misconception in problem-solving." The model often behaved like a student who has memorized formulas but lacks the conceptual depth to know when or how to apply them correctly.
  • The Paraphrase Trap: When acting as a teacher, the LLM frequently flagged correct student answers as "incorrect," only to provide a "correction" that was essentially a paraphrase of the student's original, valid logic.
  • Spatial Reasoning Deficits: The model showed a marked inability to handle 3D geometry. For instance, in problems involving fitting smaller objects into a larger container, it often relied on simple volume division rather than considering physical dimensions and divisibility.

Experimental Results

GPT-3.5 achieved an overall accuracy of approximately 66% on the MathQA dataset. Interestingly, Zero-shot prompting occasionally outperformed CoT, as the latter sometimes induced "hallucinated" reasoning paths in high-difficulty contexts.

One of the most significant findings concerned the model's pedagogical accuracy. Without a provided ground-truth rationale, the "teacher" model correctly identified the status of a student's answer only 53.96% of the time—performing only slightly better than random chance.

Comparison of pedagogical accuracy across different problem types, showing the impact of providing a ground-truth rationale.

As shown in the data, the model's accuracy improved dramatically (to ~74%) when it was provided with a reference rationale. This suggests that while LLMs struggle to autonomously detect errors, they are significantly more effective at explaining them when guided by a correct logical framework.

Conclusion and Future Directions

The study concludes that while LLMs are promising for generating educational content and fluent explanations, they lack the reliability required for autonomous tutoring in advanced mathematics. Future research should focus on Augmented Language Models (ALMs) that integrate external tools, such as Python interpreters or symbolic math engines, to handle computation and spatial logic. This would allow the LLM to focus on its primary strength: translating complex logical steps into accessible pedagogical dialogue.

Summary updated

BibTeX
@inproceedings{yen2023three,
  author = {An-Zi Yen and Wei-Ling Hsu},
  title = {{Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.201},
}
Figure