Splitting the Machine's Reasoning from the Student's Knowledge in Item Difficulty Prediction

Test makers need to know how hard a new question is before any student has seen it. One recent approach is to let a reasoning model solve the question and read difficulty off how hard it worked: the length of its written reasoning, how often it backtracked, how many verification loops it ran. This works when the model's bottleneck matches the student's. It fails when it does not. A reasoning model that has memorized pharmacology answers a hard medical licensing question in two lines, while a medical student struggles because they cannot recall the one fact the question turns on. We propose treating the model's reasoning trace as a simulator with a known blind spot, much as a driving simulator lacks real danger, and correcting that blind spot with human data from questions that have already been calibrated.

The method extracts two things from each trace. The first is a procedural signal: how much multi-step work the solution takes, which a model can simulate reasonably well. The second is the inventory of domain facts the solution depended on. Each fact gets its own human difficulty, learned from old questions that used the same fact and already have student response data. A brand-new question then inherits a knowledge-burden estimate from facts students have struggled with before, even though the question itself has no attempts. The prediction is that this repairs trace-based difficulty prediction specifically on the knowledge-heavy items where it currently collapses, and does little on logic-heavy items where it already works.

Core Claim

The disagreement between reasoning-trace difficulty and human difficulty is not noise. It has structure, and the structure is predictable. Two lines of work frame the tension. Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction (Epi2Diff) does the following:

  • It segments reasoning-model traces into cognitively labelled episodes.
  • It derives process features from those episodes.
  • It reports an 8.0% relative classification gain and a 24.4% RMSE reduction over fine-tuned encoders and supervised fine-tuning of large language models.

Its premise is that machine effort mirrors human burden. Can LLMs Estimate Student Struggles? directly undercuts that premise for knowledge-intensive material:

  • Human–model difficulty alignment averages Spearman ρ = 0.28.
  • It reaches ρ ≈ 0.41 on SAT Math but only ρ ≈ 0.13 on USMLE.
  • It reports a "Savant Rate" of 70.4%: most of the items humans find hardest are solved by more than 90% of models.

Both findings can be true at once if difficulty has two sources and the model simulates only one of them. Our claim is that an explicit decomposition recovers the human difficulty that traces miss. In this decomposition, procedural burden comes from the trace, and knowledge burden comes from human-calibrated fact difficulties indexed by the facts the trace invokes. The gain should be concentrated on "savant" items and knowledge-heavy domains.

This differs from Epi2Diff in what it does with the trace. Epi2Diff uses the trace's shape as the whole signal. We use the trace's content (which facts it leaned on) to look up human evidence that the simulator cannot produce. It also differs from the student-simulation line, which tries to make the simulator itself more human:

We do not ask the model to forget what it knows. We accept that it knows, and we borrow human difficulty for that knowledge from elsewhere.

The Simulation-Gap Lens

The ecological-validity literature supplies the framing rather than the method. Simulators reproduce surface dynamics but strip out the factor that drives real behavior. Flight simulators lack operational risk and dampen pilots' physiological load (Pilot turning behavior cognitive load analysis in simulated flight). Participants in simulations become over-bold.

Optimizing Behavioral Experiment Design with In-Silico Simulations shows the same asymmetry for language-model participants. Agent traces expose severe structural flaws in surveys but give "little value" for subtler human behavior.

The standard engineering answer to a simulator with a blind spot is not to abandon the simulator. It is to identify which variable the simulator omits, estimate that variable from real data, and add it back. Here the omitted variable is the human cost of retrieving a fact. We therefore use the model's trace for what it simulates well (stepwise work) and use real response data for what it cannot (fact retrieval).

We also borrow the field's habit of validating against a second, behavioral channel. Human response time is recorded in the BEA 2024 USMLE data and in UPSC2M. We test whether the corrected model predicts where humans hesitate, not only whether they fail.

Research Program

The pipeline has four stages.

Stage 1: Solve and segment. A reasoning model solves each item, and its trace is segmented into episodes following Epi2Diff's taxonomy. From the segmented trace we compute a procedural vector $p_i$: episode counts, transition statistics, and backtracking.

Stage 2: Extract the fact inventory. A second pass extracts the atomic domain propositions the solution relied on. An example is "loop diuretics act on the thick ascending limb." Facts are canonicalized across the item bank by clustering their sentence embeddings, so the same fact used in different questions maps to one identifier $f$.

Stage 3: Fit a difficulty model on calibrated items. On training items, which have human difficulty, we fit

$$\beta_i = g(p_i) + \mathrm{agg}_{f \in F_i}\, \delta_f + \epsilon_i,$$

The symbols are:

  • $\beta_i$ is the human difficulty of item $i$ (Rasch difficulty, or 1 − pass rate where only pass rates exist).
  • $g$ is a small regressor on procedural features.
  • $F_i$ is the item's fact set.
  • $\delta_f$ is a learned human difficulty for fact $f$, shrunk toward a prior predicted from the fact's text so that facts seen once do not overfit.
  • $\mathrm{agg}$ is either a sum or a smooth maximum.

The choice of aggregator tests a classical psychometric question. Under a sum, facts trade off against each other, which corresponds to a compensatory model. Under a smooth maximum, the hardest required fact dominates, which corresponds to a conjunctive model. We expect the conjunctive form to fit knowledge-heavy exams better.

Stage 4: Predict new items. A new item gets its trace and its fact set. Known facts use their learned $\delta_f$, and unseen facts use the text-based prior. No student attempts on the new item are needed.

Analyses. The program has two analytic pieces beyond prediction accuracy.

The first is a gap diagnostic. We take the residual of a trace-only predictor and test whether it concentrates on savant items, defined as in Li et al.: items that are hard for humans but solved by most models. We then test whether the knowledge term absorbs that residual.

The second is an ecological check. The same decomposition, with response time as the target, should show that knowledge burden predicts human latency on items where the trace is short. This is the machine analogue of the over-boldness pattern in human simulator studies: fast on exactly the items where real people slow down.

Baselines

Dummy regressor. The naive reference point predicts the training-set mean difficulty. It matters because on large real exams text-only methods barely beat it. UPSC2M reports that an MLP on frozen text embeddings reduces RMSE by only 7.1% and MAE by 9.2% relative to a dummy regressor. All gains will be reported as relative RMSE reduction against this mean predictor on identical splits.

Amortized text-embedding regression. This is the method of Reliable and Efficient Amortized Model-based Evaluation: a linear map from frozen Llama-3.1-8B embeddings to Rasch difficulty. We include the UPSC2M MLP variant of it. This is the strongest cheap cold-start competitor. It sees exactly the same input text and costs no reasoning-model calls. We run it with the same item splits, and its regularization strength is tuned on the same validation fold as our model.

Epi2Diff. This is the strongest existing method and the one our proposal must beat on its own terms. It is reimplemented with the same reasoning model, the same traces, and the same downstream regressor family. Our method therefore differs from it only by the knowledge term. The comparison number is its reported 24.4% RMSE reduction and 8.0% relative classification gain against strong baselines on its four datasets. Our own head-to-head will be on shared data.

Shared-task systems on BEA 2024 USMLE. The relevant systems are:

  • UPN-ICC: ridge regression on ChatGPT-3.5 features from simulated answering, stem mutilation, and temperature variation. It reports test RMSE 0.303, second in the task.
  • ITEC: clinical embeddings plus linguistic features in Lasso and Random Forest. It ranked 4th of 43 for difficulty, and for response time reached RMSE 24.116 against a 31.68 baseline.

Beating 0.303 RMSE on the official test split, without test-set tuning, counts as a clear win. Beating ITEC's 24.116 on response time is a secondary target.

Model-uncertainty proxies. We include choice entropy across option permutations from Can Model Uncertainty Function as a Proxy for Multiple-Choice Question Item Difficulty?. It is an unsupervised baseline, also fed into the same regressor as a supervised feature.

Proficiency-simulation pools. We include LLaSA. On DBE-KT22 it reports an F1-score of 0.321. For ASSISTments no reported number was found in the research file.

Ablations. Three ablations isolate the key ingredient.

  • Trace-only. This drops the knowledge term, recovering an Epi2Diff-like model.
  • Question-only facts. Facts are extracted by asking the model to list what the question requires, without solving it. This tests whether the facts must come from the solution path.
  • Embedding neighbors. The fact term is replaced with a k-nearest-neighbor average of human difficulty over calibrated items with similar embeddings. This tests whether "facts" are just a fancy topic similarity.

Our method beats the field only if it outperforms both Epi2Diff and embedding neighbors on knowledge-heavy items, with matched tuning budgets.

Why not just…?

Why not just concatenate text embeddings with Epi2Diff features? This is the most serious threat. An embedding of a USMLE stem plausibly encodes which pharmacology topic is involved, which may already proxy fact difficulty. The UPSC2M result says text embeddings alone carry modest signal. Even so, a combination could still match ours. The embedding-plus-Epi2Diff model is therefore run as a full baseline with identical tuning. If it ties our model on the savant subset, we will conclude that the trace-derived fact inventory adds nothing beyond topic and report that.

Why not just look up the difficulty of similar past questions? Nearest-neighbor transfer from calibrated items is a simple cold-start method and is our embedding-neighbor ablation. The case for facts is that two items with nearly identical surfaces can hinge on different facts, and two items on different topics can hinge on the same one. The decisive test is a held-out set stratified by surface similarity to training items. If fact transfer wins only where neighbors are far, the mechanism is real but narrow.

Why not just make the simulated student weaker? Persona prompts, smaller models, or a capped reasoning scratchpad could induce human-like failure, in the spirit of the working-memory account in Working Memory Identifies Reasoning Limits in Language Models. The evidence in the file argues against suppression as a route to fixing knowledge gaps:

  • Xiao et al. find that persona prompting overshoots inconsistently.
  • Li et al. find that low-proficiency personas leave alignment at ρ = 0.28 and that scaling does not help.
  • Zotos et al. find that larger models flatten the uncertainty signal.

Limiting working memory plausibly targets procedural difficulty, not recall of facts. We include a capped-scratchpad variant on SAT Math and USMLE. We expect it to help on the former and not the latter, the mirror image of our method.

Why not just ask the model to rate difficulty directly? The observer-view results in Li et al. already show weak alignment. The model's metacognitive self-error prediction has AUROC 0.56. We include direct rating only as a reference point.

Proposed Experiments

Primary testbeds. The main testbeds are UPSC2M and the BEA 2024 USMLE set.

  • UPSC2M has 8,973 questions across seven subjects with pass rates and response times from about two million attempts. Its size gives facts enough recurrence to estimate $\delta_f$ reliably.
  • The BEA 2024 USMLE set has 667 items with difficulty and response time, and is the domain where trace alignment collapses.

Contrast and secondary sets.

  • We use the SAT Math, SAT Reading, and Cambridge portions of the Li et al. suite, subject to data availability. SAT Math is the logic-heavy contrast, where we predict little gain.
  • We use the Biopsychology set from Zotos et al., which has about 268 students per item and is unpublished, so the reasoning model cannot have memorized it.
  • We use NAEP items from Xiao et al. for grade-level variation.

Splits and metrics. All evaluations use item-level cold-start splits: no test item has any attempts in training. Splits are additionally stratified by fact novelty. Metrics are RMSE and Spearman ρ against human difficulty, with RMSE also reported for response time.

Decisive checks. Three checks decide whether the proposal holds.

The first is the domain interaction. Our gain over trace-only should be large on USMLE and UPSC2M knowledge subjects and near zero on SAT Math. A uniform gain would suggest the knowledge term is just extra capacity rather than a correction for a specific blind spot.

The second is the savant subset. The trace-only residual should be concentrated on savant items, and the knowledge term should shrink it. Concretely, Spearman ρ on USMLE should move clearly above the 0.13 Li et al. report for direct model alignment.

The third is the latency dissociation. Among items with short traces, human response time should rise with predicted knowledge burden. If it does not, the knowledge term captures something other than human retrieval cost, and the simulation-gap interpretation fails even if accuracy improves.

Main risk. Fact sparsity on the 667-item USMLE set may leave most $\delta_f$ at their text prior. We will report how often facts recur and how performance varies with recurrence, and treat UPSC2M as the main test of the mechanism.

Workshop authors to connect with