Introduction
Large Language Models (LLMs) are often presented as a panacea for the administrative and pedagogical burdens of modern education. From automated grading to personalized feedback, the potential is vast. However, a significant gap exists between laboratory success and classroom deployment. This paper synthesizes a systematic scoping review of 118 studies to evaluate the current state of LLMs in education through the lenses of practicality and ethics.
Methodology
Following the PRISMA protocol, the researchers analyzed 118 peer-reviewed publications from 2017 to 2022. The evaluation focused on two primary dimensions:
- Practicality: Assessed via Technology Readiness Levels (TRL) and a check for replicability (availability of code and data).
- Ethicality: Assessed via a Transparency Index (ranging from expert-only transparency to stakeholder transparency) and a review of privacy, equality, and beneficence considerations.
The Practicality Gap: The "Lab-Only" Trap
A critical finding of the review is the prevalence of the "Lab-Only" trap. While research is prolific, over 75% of the analyzed studies remain at TRL-2 (Applied Research), indicating they are far from being ready for real-world application. Only 6% of the studies (n=7) conducted validation in authentic learning environments.
Furthermore, a replicability crisis looms over the field. Two-thirds (n=75) of the reviewed studies failed to release both their original code and the datasets used. This lack of open data creates a "black box" environment that prevents other researchers from validating or building upon existing work, significantly hindering the technological readiness of the sector.
The Dominance of Legacy Models
Despite the current cultural focus on generative models like GPT-4, 92% of the reviewed research focused on older, encoder-only models like BERT. These models often require significant manual effort for fine-tuning. The transition to state-of-the-art (SOTA) models, which could potentially achieve better results with zero-shot prompting, remains underrepresented in the empirical literature of the period studied.
Ethical and Transparency Challenges
Ethics are frequently treated as an afterthought in educational AI research. Privacy concerns were rarely addressed, and only nine studies provided descriptive data regarding the demographic distribution (gender/ethnicity) of their samples. This omission makes it nearly impossible to audit these systems for algorithmic bias, a critical requirement for tools used in diverse classrooms.
Transparency remains a significant hurdle. Most innovations reached only Tier 1 transparency, meaning the systems are only understandable to AI experts. For a teacher or a parent to trust an AI-graded essay or a dropout prediction, the field must move toward Tier 3 transparency, where the underlying logic is interpretable by non-experts.
A Taxonomy of Educational Use Cases
The review identified 53 distinct use cases for LLMs, categorized into nine functional domains:
- Profiling and Labelling: e.g., forum post classification.
- Detection: e.g., identifying student confusion or off-task behavior.
- Assessment and Grading: e.g., automated essay scoring.
- Teaching Support: e.g., intelligent Q&A systems.
- Prediction: e.g., identifying students at risk of dropping out.
- Knowledge Representation: e.g., building automated knowledge graphs.
- Feedback: e.g., providing real-time hints during problem-solving.
- Content Generation: e.g., generating multiple-choice questions (MCQs).
- Recommendation: e.g., suggesting personalized learning resources.
Quantitatively, while LLMs excel at simple classification (F1 scores > 0.90), their performance drops significantly on complex tasks like detecting student confusion or grading out-of-sample short answers.
Conclusion and Future Directions
The field of LLMs in education is currently characterized by high potential but low readiness. To move beyond the "lab phase," the authors suggest three key shifts:
- Leverage SOTA Models: Move from intensive fine-tuning of older models to exploring the zero-shot capabilities of modern generative AI.
- Commit to Open Science: Solve the replicability crisis by open-sourcing models and datasets.
- Human-Centered Design: Involve students and teachers throughout the development process to ensure ethical and practical alignment.
Addressing the transparency gap and language bias (as 95% of studies focused on English) is essential for building the trust and equity necessary for these tools to enter the classroom safely.
Summary updated