Abstract
Algebra is often challenging to teach and learn. While large language models (LLMs) are gaining attention in education, their role in algebra remains unclear. This systematic review, guided by the PRISMA framework, analyzed 21 studies from 532 records. Findings revealed that LLMs were mostly used on secondary data, for assessment, with ChatGPT, using zero-shot prompts but struggled with complex reasoning. This review offers theoretical insights and practical guidance for effectively applying LLMs in algebra education.
Introduction
Algebra serves as a critical "bridging subject" in mathematics, yet students frequently struggle with its abstract symbols and the transition from basic arithmetic. While Large Language Models (LLMs) are being rapidly integrated into general education for tasks like feedback generation and lesson planning, their specific efficacy within the rigorous, rule-based domain of algebra remains under-explored. This systematic review synthesizes existing research to map the landscape of LLM applications in algebra, identifying measurable benefits, persistent challenges, and evaluation methodologies.
Methodology
The researchers conducted a systematic literature review following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) framework. They queried six major databases—including ACM Digital Library, IEEE Xplore, and Web of Science—using the search string: (“large language model” OR LLM) AND algebra AND education.
From an initial pool of 532 records, 21 empirical, peer-reviewed studies were selected for final analysis. The team applied thematic content analysis to extract data regarding educational levels, application settings, prompting methods, and performance outcomes.
Analysis of LLM Applications in Algebra
Data Utilization and Benchmarking
As a systematic review, this study analyzes how existing datasets were utilized across the literature. Approximately half of the reviewed studies utilized existing math datasets to benchmark LLM performance rather than real-time student data. Key benchmarks included MathVista, used to compare 12 different LLMs, and various university-level math problem sets used to test models like OpenAI’s Codex.
The "Researcher" Bias
Interestingly, researchers were the primary users of LLMs in 17 of the 21 studies. This suggests that LLM application in algebra is still largely in an exploratory, lab-based phase rather than being a standard tool in the hands of students or teachers.
Prompting Techniques and the Performance Gap
The most common prompting technique identified was "zero-shot" (providing a task without examples). However, the review highlights that providing task-related information (few-shot prompting) can significantly improve outcomes, in some cases reducing failure rates from 84% to 20%. This underscores the importance of prompt engineering in mathematical contexts.
Results and Benchmarks
Accuracy and Reliability
- Model Performance: OpenAI’s Codex demonstrated the ability to solve 81% of university-level math problems. In a broader comparison of 12 models, GPT-4V achieved the highest accuracy on math datasets at 49.9%.
- Reliability Issues: Reliability remains a major hurdle; for instance, GPT-3.5 was found to generate misleading or incorrect feedback for 32% of algebra problems.
- Human Comparison: Despite their speed, LLMs generally performed below the level of an average graduate student in complex algebraic tasks.
Educational Context and Reach
Research is evenly split between K-12 and higher education, showing that the challenges of algebra instruction are universal across age groups. While LLMs might fail at complex proofs, their greatest strength in algebra education currently lies in personalized learning—providing context-specific hints and feedback that lead to measurable learning gains compared to traditional control groups.
Discussion and Future Work
The review identifies several critical roadblocks for the field:
- The Reasoning Wall: LLMs still struggle with "complex reasoning," particularly tasks requiring formal proofs or multi-step computations. The symbolic nature of algebra appears to be a specific hurdle for current autoregressive models, which perform significantly worse in algebra than in statistics or general arithmetic.
- Lack of Standardization: There is currently no comprehensive, consistent framework for evaluating LLM performance in math, making it difficult to compare results across different studies.
- Generalizability: Because many studies rely on secondary data, there is a need for more "in-the-wild" research at various educational levels to see how these tools perform in actual classrooms.
- Prompt Engineering: Future work should move away from generic zero-shot prompts toward "tailored prompting" methods specifically designed for algebraic logic.
Conclusion
This systematic review highlights that while LLMs like ChatGPT show significant potential for personalized tutoring and automated problem-solving, they currently struggle with complex mathematical reasoning and symbolic proofs. Moving forward, the field requires more standardized evaluation frameworks and a shift toward classroom-based research to truly understand the impact of LLMs on algebra education.
Summary updated