Introduction
In the landscape of modern education, adaptive learning systems generate a vast "firehose" of raw data, tracking every click, response time, and accuracy metric. While Large Language Models (LLMs) like GPT-4 have shown immense potential in providing qualitative feedback, they often struggle to process these voluminous, noisy logs directly. Raw quantitative data lacks the pedagogical context required to generate encouraging and actionable insights for students.
This paper presents a "tag-based" framework designed to bridge the gap between raw educational data and LLM-generated feedback. By transforming complex student performance logs into a structured set of descriptive tags, the system enables ChatGPT to produce personalized, constructive feedback reports that maintain a professional and motivating tone.
Methodology
Data Source and Preprocessing
The study utilized data from a three-dimensional adaptive learning system implemented in Shanghai primary schools. This system tracks performance (correctness), behavior (time spent and difficulty levels), and metadata (knowledge categories and cognitive ability levels).
To manage this data, the researchers simplified over 50 knowledge categories into six primary groups, such as "Geometric Thinking Skills" and "Arithmetic Operations." Noise, including accidental clicks or zero-second attempts, was filtered to ensure data integrity.
The Tag Annotation System
The core innovation of this research is the use of Tags as a semantic bridge. Instead of feeding raw numbers to the LLM, the system assigns students specific tags across three categories:
- Basic Analysis: Combines accuracy and speed (e.g., "Correctly but completed slowly on difficult questions").
- Knowledge Category Analysis: Highlights domain-specific strengths or weaknesses (e.g., "Outstanding in algebraic equations").
- Ability Analysis: Maps performance to cognitive skills (e.g., "Challenged by logical reasoning").
In total, 34 tags were developed to provide the LLM with the necessary context to be "smart" without the noise that typically leads to model hallucination.
Structured Prompt Engineering
These tags are processed by GPT-4 using a highly structured prompt. The model is instructed to write in the second person ("You") and organize the report into distinct sections: Overview, Basic Analysis, Knowledge Analysis, Ability Analysis, Recommendations, and a Summary. To ensure professional consistency and reduce variability, the model's temperature was set to 0.4.
Evaluation and Results
The system's effectiveness was evaluated by 32 mathematics teachers who reviewed the generated reports across five dimensions on a scale of 0–10.
Key Findings
- Motivation Effect: This dimension scored the highest (median ~8.0), indicating that the AI successfully adopted a constructive tone that encourages student growth rather than merely highlighting failure.
- Understanding and Organization: Both metrics performed well (median ~7.5), confirming that the tag-based approach accurately captured student profiles and presented them in a logical, readable format.
- Practicality: Teachers found the AI-generated suggestions to be applicable to real-world classroom needs.
- Clarity: This was the lowest-scoring dimension (average 6.59). Feedback indicated that the language generated by the LLM was occasionally too complex for primary-aged students, requiring teacher interpretation.
Discussion and Future Work
The research demonstrates that tags act as an effective form of "dimensionality reduction," preserving pedagogical meaning while simplifying data for LLM consumption. By shifting the focus from a "post-mortem" of errors to a roadmap for growth (using phrases like "Challenged by..." instead of "Failed at..."), the system provides a more holistic and encouraging view of the student's journey.
Future work will focus on three primary areas:
- Language Optimization: Adjusting the prompt to ensure the reading level is appropriate for younger children.
- Visual Integration: Incorporating charts and visual data summaries alongside the text-based feedback.
- Scaling: Expanding the testing phase to include larger sample sizes and diverse subject matters beyond mathematics.
Reference: Zhou, Y., et al. (2024). A Study on Educational Data Analysis and Personalized Feedback Report Generation Based on Tags and ChatGPT. Proceedings of the 28th Global Chinese Conference on Computers in Education (GCCCE 2024).
Summary updated