CodeGENCAT: Generative Computerized Adaptive Testing for Open-ended Coding Problems

Wanyong Feng, Alexander Scarlatos, Ruochen Sun, Andrew LanarXiv 2026 · arXiv

TL;DR

Traditional Computerized Adaptive Testing (CAT) reduces student responses to simple binary "correct/incorrect" labels, throwing away rich diagnostic data. CodeGENCAT fixes this for programming education by using a Generative Item Response Theory (GIRT) model to predict actual student code submissions. By evaluating how a student might write code, the framework selects significantly more informative questions, boosting knowledge estimation accuracy by up to 4.32% in the early stages of testing.

What Problem Does This Paper Address?

Computerized Adaptive Testing (CAT) powers major standardized exams by dynamically selecting questions tailored to a student's ability level. Most CAT systems rely on Item Response Theory (IRT), which treats student answers as binary: either correct (1) or incorrect (0).

In programming education, this binary abstraction is a massive bottleneck. Two students might both fail the same coding challenge, but one might have made a minor syntax typo while the other completely misunderstood the underlying algorithm. Treating both as simply "incorrect" ignores crucial context about their actual conceptual gaps. The authors tackle this by moving beyond binary correctness, leveraging the rich textual information hidden inside open-ended code submissions to better assess student knowledge.

Methods

CodeGENCAT introduces a loop that alternates between predicting student code and updating knowledge estimates. The framework relies on two core innovations:

1. Generative Item Response Theory (GIRT)

Instead of predicting a probability of correctness, GIRT predicts the actual sequence of code tokens a student will write.

  • Knowledge Representation: A student's latent knowledge is mapped to explicit mastery levels across various Knowledge Components (KCs), like loops or arrays.
  • LLM-Driven Generation: These mastery levels are transformed into continuous embeddings and injected into an LLM prompt (using Llama-3.2-Instruct as a backbone). The LLM then simulates the student's code response.
  • Alignment Training: To ensure the LLM accurately reflects student ability, it undergoes two training stages: Supervised Fine-Tuning (SFT) with an auxiliary alignment loss, followed by Direct Preference Optimization (DPO). DPO forces the model to generate logically flawed code when conditioned on low mastery levels, and clean code for high mastery.

2. Question-Selection Algorithms

At each test step, the system samples $N$ possible code responses from the GIRT model for each candidate question and evaluates them using three novel strategies:

  • Uncertainty-Based: Chooses the question where the predicted correctness score of the simulated code is closest to 0.5.
  • Diversity-Based: Uses CodeBERT to measure the semantic diversity of the generated code samples. High diversity implies the model is highly uncertain about how the student will respond, making the question informative.
  • Fisher-Information-Based: Calculates the trace of the Fisher information matrix based on the log-likelihood gradient of the GIRT model with respect to the student's latent knowledge.
Figure 1: Overview of one CodeGENCAT cycle with Diversity as the question selection algorithm.

Datasets

The authors evaluated their framework on two real-world programming education datasets:

  1. CodeWorkout: A public dataset containing Java code submissions from 246 students tackling 50 introductory computer science problems (mapped to 18 distinct KCs).
  2. ProgFeed: A newly collected dataset consisting of Python code submissions from 374 students across 42 introductory programming questions (mapped to 10 distinct KCs).

Key Insights and Ideas

  • The Necessity of DPO: Standard supervised fine-tuning (SFT) suffers from mode collapse; the LLM tends to generate correct code regardless of the student's input mastery level. DPO is essential to force the model to generate realistic, buggy code for struggling students.
  • "How" Matters More Than "If": Predicting how a student will fail provides a much stronger signal for question selection than simply predicting whether they will fail. This is why generative modeling beats traditional binary IRT.
  • Temporal Strategy Shifts: Maximizing response diversity is highly effective in the very early stages of a test when the system relies heavily on population priors. Later on, gradient-based Fisher Information becomes superior as the student's knowledge profile sharpens.

Key Results

CodeGENCAT consistently outperformed strong CAT baselines (including standard 1PL IRT and LLM-agent frameworks like LACAT), particularly in the early stages of testing (steps 1 to 7).

  • Accuracy Boost: On the CodeWorkout dataset, the Diversity-based selection algorithm achieved a 4.32% AUC improvement over the strongest baseline at the very first test stage.
  • Consistency: The Information-based variant consistently maintained a performance edge across later test stages on both datasets.
Table 1: 5-fold averaged knowledge estimate performance (AUC and Accuracy) on both datasets across varying test lengths (t). CodeGENCAT consistently outperforms baselines, particularly in the early stages (t = 1 to 7), with Diversity, and late stages (t = 10 to 20) with Information on the CodeWorkout dataset. CodeGENCAT consistently outperforms baselines for most time steps with Information on the ProgFeed dataset. * denotes statistically significant improvement over the best baseline using a paired sample t-test (p < 0.05). For AUC, the best baselines are 1PL IRT and NCAT, and for accuracy, the best baselines are NCAT and LACAT, for CodeWorkout and ProgFeed respectively. Best results are in bold and second best are underlined.
  • Test Security: The Uncertainty-based variant achieved the lowest test overlap rate (25.33%), demonstrating that the framework naturally balances test efficiency with item security requirements.

Limitations and Future Work

Despite its strong performance, CodeGENCAT has a few notable limitations:

  • Computational Overhead: Generating multiple code samples ($N=10$) for every candidate question at every step is computationally heavy and introduces latency challenges for real-time deployment.
  • Domain Specificity: The current evaluation is restricted to introductory computer science. Scaling it to other open-ended fields (like mathematical reasoning or essay writing) remains an open empirical challenge due to data scarcity.
  • Metadata Reliance: The framework depends on expert-annotated Knowledge Components, limiting its immediate scalability to subjects where structured metadata is unavailable.

Future work will focus on optimizing response sampling to reduce inference latency and exploring automated KC generation to eliminate the need for human annotation.

Summary updated

Abstract

Existing Computerized Adaptive Testing (CAT) frameworks typically select questions based on the predicted likelihood that the student will answer correctly. This design ignores information contained in students' open-ended responses, especially in domains such as programming education, where code structures and bugs contain rich information on student knowledge. In this work, we propose \textbf{Code} \textbf{GEN}erative \textbf{CAT} (\textbf{CodeGENCAT}), a generative CAT framework that selects questions using predicted student code responses. First, we develop a Generative Item Response Theory (GIRT) model that generates code responses conditioned on estimated student knowledge, trained with supervised fine-tuning followed by direct preference optimization for knowledge-response alignment. Second, we introduce three question-selection algorithms that measure uncertainty, coding style diversity, and information from predicted student code responses. Experiments on two real-world programming education datasets show that CodeGENCAT outperforms all CAT baselines, achieving an AUC improvement of up to 4.32\% over the strongest baseline in the early stages of adaptive testing.
BibTeX
@misc{feng2026codegencat,
  author = {Wanyong Feng and Alexander Scarlatos and Ruochen Sun and Andrew Lan},
  title = {{CodeGENCAT: Generative Computerized Adaptive Testing for Open-ended Coding Problems}},
  year = {2026},
  eprint = {2602.20020},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.20020},
}

Loading related content...

Figure