A Study on Educational Data Analysis and Personalized Feedback Report Generation Based on Tags and ChatGPT

Yizhou Zhou, Mengqiao Zhang, Yuan-Hao Jiang, Xinyu Gao, Naijie Liu, Bo JiangarXiv 2025 · DOI · arXiv

Introduction

In the landscape of modern education, adaptive learning systems generate a vast "firehose" of raw data, tracking every click, response time, and accuracy metric. While Large Language Models (LLMs) like GPT-4 have shown immense potential in providing qualitative feedback, they often struggle to process these voluminous, noisy logs directly. Raw quantitative data lacks the pedagogical context required to generate encouraging and actionable insights for students.

This paper presents a "tag-based" framework designed to bridge the gap between raw educational data and LLM-generated feedback. By transforming complex student performance logs into a structured set of descriptive tags, the system enables ChatGPT to produce personalized, constructive feedback reports that maintain a professional and motivating tone.

Methodology

Data Source and Preprocessing

The study utilized data from a three-dimensional adaptive learning system implemented in Shanghai primary schools. This system tracks performance (correctness), behavior (time spent and difficulty levels), and metadata (knowledge categories and cognitive ability levels).

The adaptive learning system interface used to collect student performance and behavioral data.

To manage this data, the researchers simplified over 50 knowledge categories into six primary groups, such as "Geometric Thinking Skills" and "Arithmetic Operations." Noise, including accidental clicks or zero-second attempts, was filtered to ensure data integrity.

The Tag Annotation System

The core innovation of this research is the use of Tags as a semantic bridge. Instead of feeding raw numbers to the LLM, the system assigns students specific tags across three categories:

  1. Basic Analysis: Combines accuracy and speed (e.g., "Correctly but completed slowly on difficult questions").
  2. Knowledge Category Analysis: Highlights domain-specific strengths or weaknesses (e.g., "Outstanding in algebraic equations").
  3. Ability Analysis: Maps performance to cognitive skills (e.g., "Challenged by logical reasoning").

In total, 34 tags were developed to provide the LLM with the necessary context to be "smart" without the noise that typically leads to model hallucination.

Structured Prompt Engineering

These tags are processed by GPT-4 using a highly structured prompt. The model is instructed to write in the second person ("You") and organize the report into distinct sections: Overview, Basic Analysis, Knowledge Analysis, Ability Analysis, Recommendations, and a Summary. To ensure professional consistency and reduce variability, the model's temperature was set to 0.4.

Evaluation and Results

The system's effectiveness was evaluated by 32 mathematics teachers who reviewed the generated reports across five dimensions on a scale of 0–10.

Teacher evaluation results across five key dimensions: Understanding Level, Practicality, Motivation Effect, Clarity, and Organizational Structure.

Key Findings

  • Motivation Effect: This dimension scored the highest (median ~8.0), indicating that the AI successfully adopted a constructive tone that encourages student growth rather than merely highlighting failure.
  • Understanding and Organization: Both metrics performed well (median ~7.5), confirming that the tag-based approach accurately captured student profiles and presented them in a logical, readable format.
  • Practicality: Teachers found the AI-generated suggestions to be applicable to real-world classroom needs.
  • Clarity: This was the lowest-scoring dimension (average 6.59). Feedback indicated that the language generated by the LLM was occasionally too complex for primary-aged students, requiring teacher interpretation.

Discussion and Future Work

The research demonstrates that tags act as an effective form of "dimensionality reduction," preserving pedagogical meaning while simplifying data for LLM consumption. By shifting the focus from a "post-mortem" of errors to a roadmap for growth (using phrases like "Challenged by..." instead of "Failed at..."), the system provides a more holistic and encouraging view of the student's journey.

Future work will focus on three primary areas:

  1. Language Optimization: Adjusting the prompt to ensure the reading level is appropriate for younger children.
  2. Visual Integration: Incorporating charts and visual data summaries alongside the text-based feedback.
  3. Scaling: Expanding the testing phase to include larger sample sizes and diverse subject matters beyond mathematics.

Reference: Zhou, Y., et al. (2024). A Study on Educational Data Analysis and Personalized Feedback Report Generation Based on Tags and ChatGPT. Proceedings of the 28th Global Chinese Conference on Computers in Education (GCCCE 2024).

Summary updated

BibTeX
@misc{zhou2025study,
  author = {Yizhou Zhou and Mengqiao Zhang and Yuan-Hao Jiang and Xinyu Gao and Naijie Liu and Bo Jiang},
  title = {{A Study on Educational Data Analysis and Personalized Feedback Report Generation Based on Tags and ChatGPT}},
  year = {2025},
  eprint = {2501.06819},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.06819},
}
Figure