Learning Code-Edit Embedding to Model Student Debugging Behavior

Hasnain Heickal, Andrew S. LanarXiv 2025 · DOI · arXiv

Abstract

This paper introduces a novel encoder-decoder framework designed to model the iterative process of student debugging. By shifting the focus from static code states to code-edit embeddings, the model learns to represent the semantic transitions between submissions. Using test-case results as a semantic anchor in a contrastive learning objective, the system provides personalized, incremental feedback that preserves student coding style while avoiding the "spoiler" effect common in large language models.

Introduction

In computer science education, feedback is a double-edged sword. Traditional auto-graders often provide "binary" feedback—your code passed or it didn't—which leaves struggling students guessing. On the other hand, modern Large Language Models (LLMs) like GPT-4 are often too helpful; they tend to provide the entire corrected solution, bypassing the "desirable difficulty" necessary for a student to actually learn how to debug.

The authors identify a major gap: we lack tools that understand the process of debugging. Most existing models look at a single snapshot of code. However, debugging is iterative. To provide truly personalized support, a system needs to understand the trajectory of a student's edits—how they move from a broken state to a slightly less broken state—while preserving their unique coding style.

Methods

The authors propose an encoder-decoder framework based on CodeT5. The core innovation is the shift from embedding code to embedding code-edits.

  1. The Edit Encoder: Instead of just looking at Code A or Code B, the model takes the embeddings of two consecutive submissions and passes their difference through a Multi-Layer Perceptron (MLP) to create a "code-edit embedding."
  2. Contrastive Learning via Test Cases: To teach the model what a "meaningful" edit looks like, the authors use test cases as a semantic anchor. If two different students make different edits that result in the exact same change in test-case passes/failures, the model is trained to see those edits as "similar."
  3. Multi-Objective Loss: The model is trained on three goals simultaneously: Contrastive Loss (grouping similar debugging behaviors), Reconstruction Loss (ensuring the decoder can turn an embedding back into code), and Regularization Loss (aligning the edit space with the original code space).
Architecture of the code-edit embedding model, showing the flow from code pairs to edit embeddings and multi-objective loss calculation.

This allows the model to take a student's current (buggy) code and a "target" test-case mask, then retrieve a realistic edit from a database of historical student behavior to generate a personalized hint.

Datasets and Evaluation

The study utilizes the 2nd CSEDM Data Challenge dataset, focusing on 17 programming problems with accompanying test cases, resulting in roughly 8,000 unique code-edit pairs. The data was split by student (90/5/5) to ensure generalizability.

The model was evaluated against GPT-4o and standard CodeT5 baselines using CodeBLEU and Test-Case-Mask Adherence.

Key Results and Insights

  • Incremental Progress: The proposed model achieved an exact match for desired test-case masks 56% of the time, compared to only 2% for GPT-4o. This proves the model is far superior at providing "step-by-step" guidance rather than "spoiling" the full solution.
  • Personalization: Suggestions from this model were significantly closer to the student's original coding style than GPT-generated code.
  • Clustering Accuracy: The model successfully grouped edits by their semantic function. By clustering these edit embeddings, the authors found that students across different problems often make the same types of mistakes, such as "off-by-one" errors.
t-SNE visualization of the embedding space under different loss configurations, demonstrating how the full multi-objective loss (d) produces the most distinct semantic clusters.

Limitations and Future Work

While the model excels at logic and personalization, it occasionally produces minor syntactical errors. Future work includes using LLMs as "post-processors" to clean up syntax while keeping the logic generated by the edit-embedding model, and expanding the methodology beyond Java to other programming languages.

Summary updated

BibTeX
@misc{heickal2025learning,
  author = {Hasnain Heickal and Andrew S. Lan},
  title = {{Learning Code-Edit Embedding to Model Student Debugging Behavior}},
  year = {2025},
  eprint = {2502.19407},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.19407},
}
Figure