TL;DR
Traditional Computerized Adaptive Testing (CAT) reduces student responses to simple binary "correct/incorrect" labels, throwing away rich diagnostic data. CodeGENCAT fixes this for programming education by using a Generative Item Response Theory (GIRT) model to predict actual student code submissions. By evaluating how a student might write code, the framework selects significantly more informative questions, boosting knowledge estimation accuracy by up to 4.32% in the early stages of testing.
What Problem Does This Paper Address?
Computerized Adaptive Testing (CAT) powers major standardized exams by dynamically selecting questions tailored to a student's ability level. Most CAT systems rely on Item Response Theory (IRT), which treats student answers as binary: either correct (1) or incorrect (0).
In programming education, this binary abstraction is a massive bottleneck. Two students might both fail the same coding challenge, but one might have made a minor syntax typo while the other completely misunderstood the underlying algorithm. Treating both as simply "incorrect" ignores crucial context about their actual conceptual gaps. The authors tackle this by moving beyond binary correctness, leveraging the rich textual information hidden inside open-ended code submissions to better assess student knowledge.
Methods
CodeGENCAT introduces a loop that alternates between predicting student code and updating knowledge estimates. The framework relies on two core innovations:
1. Generative Item Response Theory (GIRT)
Instead of predicting a probability of correctness, GIRT predicts the actual sequence of code tokens a student will write.
- Knowledge Representation: A student's latent knowledge is mapped to explicit mastery levels across various Knowledge Components (KCs), like loops or arrays.
- LLM-Driven Generation: These mastery levels are transformed into continuous embeddings and injected into an LLM prompt (using Llama-3.2-Instruct as a backbone). The LLM then simulates the student's code response.
- Alignment Training: To ensure the LLM accurately reflects student ability, it undergoes two training stages: Supervised Fine-Tuning (SFT) with an auxiliary alignment loss, followed by Direct Preference Optimization (DPO). DPO forces the model to generate logically flawed code when conditioned on low mastery levels, and clean code for high mastery.
2. Question-Selection Algorithms
At each test step, the system samples $N$ possible code responses from the GIRT model for each candidate question and evaluates them using three novel strategies:
- Uncertainty-Based: Chooses the question where the predicted correctness score of the simulated code is closest to 0.5.
- Diversity-Based: Uses CodeBERT to measure the semantic diversity of the generated code samples. High diversity implies the model is highly uncertain about how the student will respond, making the question informative.
- Fisher-Information-Based: Calculates the trace of the Fisher information matrix based on the log-likelihood gradient of the GIRT model with respect to the student's latent knowledge.
Datasets
The authors evaluated their framework on two real-world programming education datasets:
- CodeWorkout: A public dataset containing Java code submissions from 246 students tackling 50 introductory computer science problems (mapped to 18 distinct KCs).
- ProgFeed: A newly collected dataset consisting of Python code submissions from 374 students across 42 introductory programming questions (mapped to 10 distinct KCs).
Key Insights and Ideas
- The Necessity of DPO: Standard supervised fine-tuning (SFT) suffers from mode collapse; the LLM tends to generate correct code regardless of the student's input mastery level. DPO is essential to force the model to generate realistic, buggy code for struggling students.
- "How" Matters More Than "If": Predicting how a student will fail provides a much stronger signal for question selection than simply predicting whether they will fail. This is why generative modeling beats traditional binary IRT.
- Temporal Strategy Shifts: Maximizing response diversity is highly effective in the very early stages of a test when the system relies heavily on population priors. Later on, gradient-based Fisher Information becomes superior as the student's knowledge profile sharpens.
Key Results
CodeGENCAT consistently outperformed strong CAT baselines (including standard 1PL IRT and LLM-agent frameworks like LACAT), particularly in the early stages of testing (steps 1 to 7).
- Accuracy Boost: On the CodeWorkout dataset, the Diversity-based selection algorithm achieved a 4.32% AUC improvement over the strongest baseline at the very first test stage.
- Consistency: The Information-based variant consistently maintained a performance edge across later test stages on both datasets.
- Test Security: The Uncertainty-based variant achieved the lowest test overlap rate (25.33%), demonstrating that the framework naturally balances test efficiency with item security requirements.
Limitations and Future Work
Despite its strong performance, CodeGENCAT has a few notable limitations:
- Computational Overhead: Generating multiple code samples ($N=10$) for every candidate question at every step is computationally heavy and introduces latency challenges for real-time deployment.
- Domain Specificity: The current evaluation is restricted to introductory computer science. Scaling it to other open-ended fields (like mathematical reasoning or essay writing) remains an open empirical challenge due to data scarcity.
- Metadata Reliance: The framework depends on expert-annotated Knowledge Components, limiting its immediate scalability to subjects where structured metadata is unavailable.
Future work will focus on optimizing response sampling to reduce inference latency and exploring automated KC generation to eliminate the need for human annotation.
Summary updated