Automated Assessment of Student Self-Explanation in Code Comprehension Using Pre-Trained Language Models

Jeevan Chapagain, Vasile RusAAAI 2025 · DOI

Abstract

Self-explanation is a critical pedagogical tool in computer science education, yet manual assessment at scale remains a significant challenge. This study investigates the application of fine-tuned transformer-based language models to automate the grading of student explanations for Java code snippets. By framing the problem as a semantic similarity regression task, the researchers evaluated several pre-trained models, including BERT, CodeBERT, and SciBERT. Results indicate that SciBERT, a model trained on scientific literature, achieves the highest correlation with human expert ratings, particularly when using an "Average-of-Max" aggregation strategy for multi-sentence responses.

Introduction

In computer science education, "self-explanation"—the process where students articulate the logic of a code snippet in their own words—is a proven method for deepening comprehension. However, the labor-intensive nature of grading these free-text responses limits their use in large-scale courses. Automated Short Answer Grading (ASAG) systems face unique challenges in the programming domain, such as the mix of natural and technical language, the need for logical precision, and the high variability of correct answers. This paper explores the use of Pre-Trained Language Models (PLMs) to provide accurate, automated feedback within Intelligent Tutoring Systems (ITS).

Methodology

The researchers approached the assessment as a regression task, training models to predict a similarity score between 0 and 1 by comparing student explanations against expert references.

Model Architecture

The core architecture utilizes a transformer-based backbone with a regression head. Specifically, a fully connected layer is appended to the [CLS] token output of the pre-trained model. The entire stack is then fine-tuned on pairs of student and expert explanations.

Model architecture for regression-based similarity assessment.

The study evaluated several models:

  • General Purpose: BERT, RoBERTa, and DistilBERT.
  • Domain Specific: CodeBERT (pre-trained on code/natural language) and SciBERT (pre-trained on scientific papers).

Aggregation Strategies

To handle multi-sentence responses, three strategies were tested:

  1. Max_of_Max: The highest similarity score among all sentence pairings.
  2. Avg_of_Max: The average of the best-matching scores for each student sentence.
  3. Max_of_Avg: The highest average similarity across sentence groups.

Dataset

The evaluation utilized the SelfCode 2.0 dataset, comprising 3,019 student-generated self-explanations for 10 Java code examples. These were compared against expert explanations provided by Computer Science PhD students. Human-assigned similarity ratings (1–5 scale) served as the ground truth for training and evaluation.

Results and Discussion

The fine-tuned transformer models significantly outperformed traditional baselines such as TF-IDF, BERTScore, and Universal Sentence Encoder (USE).

Performance comparison of various models and aggregation strategies showing Pearson (P) and Spearman (S) correlations.

Key Findings

  • SciBERT Superiority: Surprisingly, SciBERT outperformed CodeBERT, achieving a Pearson correlation of 0.784 and a Spearman correlation of 0.775 using the Avg_of_Max method. This suggests that SciBERT's training on diverse scientific writing helps it bridge the gap between formal expert language and informal student phrasing.
  • Optimal Aggregation: The "Average-of-Max" strategy proved most effective, as it rewards core correct points while accounting for irrelevant or incorrect auxiliary statements.
  • Semantic Understanding: Traditional keyword-based methods like TF-IDF performed poorly (Pearson ~0.31), highlighting the necessity of deep semantic understanding provided by transformers.

Conclusion and Future Work

This research demonstrates that fine-tuned PLMs can achieve human-like consistency in grading code explanations. However, several avenues for improvement remain. Current models rely on a single expert reference and lack broader programmatic context (e.g., the overall algorithm's goal). Future work should focus on multi-expert references, cross-language generalization (e.g., Python, C++), and improving model interpretability to provide actionable feedback to students.

Summary updated

BibTeX
@inproceedings{chapagain2025automated,
  author = {Jeevan Chapagain and Vasile Rus},
  title = {{Automated Assessment of Student Self-Explanation in Code Comprehension Using Pre-Trained Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35169},
}
Figure