Language models are increasingly used to guess how hard an exam question will be for students, so that new items can be calibrated before anyone takes them. They are surprisingly bad at it. Across more than twenty frontier models and four exams, model difficulty judgments correlate with real student difficulty at only about 0.28 on average (Spearman), and the correlation falls to about 0.13 on medical licensing questions (Can LLMs Estimate Student Struggles?). That study attributes the failure to a "curse of knowledge": the model knows too much to feel what a student feels. This note tests a more specific and checkable explanation. We think much of the human difficulty signal is present in the pretrained model, which learned to imitate text written by fallible people, and that the post-training steps that make a model a confident, helpful assistant remove it. Those steps are supervised instruction tuning followed by preference optimization, meaning training that rewards answers people prefer.
The plan is to hold the model family fixed and measure the same question-level signals at each released checkpoint: the base model, the instruction-tuned model, and the preference-tuned model. The signals are how much probability the model puts on wrong answers, whether those are the same wrong answers students pick, and how much hesitation and backtracking appear in its written reasoning. If post-training erases this signal, the field has been reading difficulty from the wrong checkpoint. A cheap estimator then follows directly: read difficulty from the base model, and from how far post-training moves the model's beliefs on each item. If the signal does not change across checkpoints once accuracy is controlled, the curse-of-knowledge account stands, and alignment can be ruled out as a cause.
Core Claim
The claim has two parts, and only the first is the scientific bet. The first part is that, within one model family and at matched accuracy, agreement between the model's per-question uncertainty and human item difficulty decreases from base to instruction-tuned to preference-tuned checkpoints. The decrease should be sharpest for "attractor" items, where students converge on one specific wrong answer. The second part is practical. Features read from the base model, together with the size of the base-to-aligned shift on each item, should predict human difficulty better than features from the aligned model alone, including the strongest existing student-simulation method.
The two seed papers give the two halves of the story. Cognitive Episodes in LLM Reasoning Traces shows that sentence-level "episodes" in a reasoning model's trace (verifying, backtracking, recovering from errors) predict human difficulty. It measures only off-the-shelf post-trained models and never asks whether post-training shortens or reshapes those episodes. Psychometric Framework for Comparing Alignment-Conditioned Behavior profiles how alignment regimes shift models' personality-like behavior. Its profiles are descriptive and are not linked to any task outcome. We connect the two by asking whether the same post-training pressure that produces an agreeable, confident persona also removes the computational traces of struggle that difficulty prediction depends on.
Why We Expect This, and What Constrains It
The motivating evidence comes from work on alignment and persona fidelity. The Narcissus Hypothesis tracks 31 models from 2019 to 2025. It finds social-desirability bias rising by 0.0466 points per year, with neuroticism falling, which in practice means fewer admissions of uncertainty. Breaking the Assistant Mold finds that aligned character generators answer questions directly nearly 100% of the time. Both describe a pressure toward confident resolution. Neither measures what that pressure does to task-level uncertainty.
Three results constrain the design. The first comes from puzzle-difficulty research: Generalized Entropy and Solution Information finds that a solver given too many correct inference shortcuts correlates worse with human ratings. A good difficulty model therefore needs a solver that is fallible in the way humans are, not just a weaker solver. This is why we test whether the model chooses the same wrong answers students choose, and do not rely on overall accuracy. The second constraint is that the curse-of-knowledge study reports that scaling from GPT-4 to GPT-5-class models does not help and sometimes hurts (Can LLMs Estimate Student Struggles?). That fits our hypothesis, but it is also consistent with a pure capability story, so capability has to be controlled explicitly. The third comes from student simulation: SMART and PS² show that fine-tuning or logit blending can recover low-ability behavior. These are the methods we must beat. PS²'s low-ability model is trained on errors written by GPT-4o, an aligned model, and if our hypothesis holds, that is exactly the wrong source for human errors.
Research Program
We use open model families that release base, instruction-tuned, and preference-tuned checkpoints from the same pretraining run, plus reasoning-tuned variants where they exist. We cover at least three families and two sizes per family. Size variation is what allows capability to be separated from post-training stage.
For each checkpoint and each multiple-choice item, we extract four signals. The first is the probability distribution over answer options, scored by log-likelihood in a fixed few-shot format that works the same way for base and chat models. The second is the KL divergence between that distribution and the human response distribution over options, where per-option student data exist. The third is the match between the model's most probable wrong option and the students' most popular wrong option. The fourth is a set of episode features from sampled chain-of-thought traces, coded with the Epi2Diff taxonomy from the seed paper. Base models get few-shot reasoning prompts so that all stages produce traces in comparable formats.
We also run the TRAIT-style personality inventory (TRAIT) on each checkpoint, which yields a per-stage measure of agreeableness and social desirability. This is a secondary use of the second seed paper. Across families, we test whether a larger social-desirability shift during post-training predicts a larger loss of difficulty signal. If it does, the persona effect and the difficulty effect share a cause. If it does not, they are separate side effects.
The analysis has three steps. The stage effect is estimated as the Spearman correlation between each checkpoint's uncertainty and human difficulty. We report it both raw and as a partial correlation controlling for the checkpoint's own correctness on each item, and we repeat the comparison on item subsets where base and aligned checkpoints have equal accuracy. The mechanism step asks which signals disappear: option entropy, human-distractor match, or specific episode types such as hesitation and backtracking. The estimator step fits one small regressor, either ridge or gradient boosting, on three feature sets: aligned-model features, base-model features, and base features plus the per-item shift between base and aligned option distributions. The shift should be informative because post-training moves confidence the most on items that the base model, like the students, found ambiguous.
Baselines
The naive reference is an aligned chat model asked directly to rate item difficulty, as in the observer protocol of Can LLMs Estimate Student Struggles?, which reports an average Spearman correlation of 0.28, 0.41 on SAT Math, and about 0.13 on USMLE. We rerun this protocol on our item splits with the same aligned checkpoints we analyze, so the stage comparison and the baseline share models. A persona variant ("answer as a struggling student") is run in the same way.
The strongest existing method on multiple-choice data with human error data is PS², which blends the logits of an expert model and a model fine-tuned on synthetic errors. It reports a Pearson correlation of 0.4412 on EEDI against 0.3094 for a vanilla LLM (PS²). We reimplement it on our EEDI split with its published model sizes and give it the same number of hyperparameter trials as our regressor. To beat it, our base-plus-shift estimator must exceed the rerun PS² correlation, with a bootstrap confidence interval over items excluding zero difference. As a sanity anchor, it should also approach or exceed the 0.4412 that PS² reports. For open-ended reading items, SMART reports a Pearson correlation of 0.67 on Smarter Balanced against 0.49 for ModernBERT (SMART). Our method is designed for multiple-choice items, so SMART serves only as context unless we extend the method to open responses.
On USMLE items from the BEA 2024 shared task (466 training, 201 test), the reference points are an RMSE of 0.253 from the linguistic-feature system in Item Difficulty and Response Time Prediction … USMLE Items and a best difficulty MSE of 0.0500 from fine-tuned BERT with LLM-answer features in UnibucLLM. We use the official split, so these figures are directly comparable. Epi2Diff applied to an aligned reasoning model is the trace-based baseline. Its paper reports only relative gains (8.0% in classification, 24.4% lower RMSE) on datasets the abstract does not name, so no reported absolute result was found on Lacuna for Epi2Diff on EEDI or USMLE, and we rerun it ourselves. A probe on aligned-model hidden states trained on human labels, following The LLM Already Knows, is the white-box baseline.
The ablation that isolates the key ingredient keeps the regressor, prompts, and items fixed and changes only the checkpoint the features are read from. If the base checkpoint does not beat its own aligned descendant under this ablation, the core claim fails regardless of the headline numbers.
Why not just…?
Why not just prompt the aligned model to role-play a weak student? This has already been tried at scale. Persona prompting leaves accuracy high, and models predict their own errors with an AUROC of 0.56 (Can LLMs Estimate Student Struggles?). PS² reports that lower-proficiency prompts sometimes outscore higher-proficiency ones. We still include persona prompting as a baseline at every checkpoint, which also shows whether role-play works better on base models.
Why not just raise the sampling temperature on the aligned model? This is the most serious threat. If post-training only sharpens a distribution whose ranking is unchanged, then rescaling temperature would recover everything and the "erasure" would be cosmetic. We test this directly. For each aligned checkpoint we fit a single temperature that matches its average option entropy to the base model's, and then compare per-item rankings and human-distractor match. Temperature rescaling preserves the order of the options, so it cannot change which wrong answer is most probable. If base models match student distractors more often and that advantage survives rescaling, the information has actually been lost. If it does not survive, we will report that the effect is calibration only and that the cheap fix is enough.
Why not just conclude the base model looks human because it is less capable? A weaker model fails more often, and its failures will correlate with human failures for that reason alone. This is why every stage comparison is reported as a partial correlation controlling for the checkpoint's own correctness, on accuracy-matched subsets, and against a smaller aligned model from the same family whose accuracy is close to that of the larger base model. If the small aligned model tracks human difficulty as well as the large base model, capability explains the effect and our account is wrong.
Why not just fine-tune on human response data, as SMART and PS² do? When data are plentiful, that may win. Our claim is about the cold-start setting, where new items have no student responses, and about where the fine-tuning should begin. As a follow-up, we initialize PS²'s low-ability model from the base checkpoint and from the aligned one. If the base initialization helps, that is further evidence for the hypothesis.
Proposed Experiments
The main datasets are EEDI, which has per-option student response rates, and the BEA 2024 USMLE items. SAT Math, SAT Reading, and Cambridge items are added if the release accompanying the curse-of-knowledge study makes them available. Domain differences sharpen the prediction: knowledge-heavy USMLE, where aligned models diverge most at ρ≈0.13, should show the largest base-versus-aligned gap, and logic-heavy SAT Math should show the smallest.
Three results decide the question. First, does the partial correlation between uncertainty and human difficulty drop monotonically across post-training stages in at least two of three families, after controlling for accuracy? Second, do base checkpoints match students' modal wrong answer on EEDI more often than temperature-rescaled aligned checkpoints? Third, does the base-plus-shift estimator beat the rerun PS² on EEDI and the 0.253 RMSE / 0.0500 MSE reference points on the USMLE test split? Mechanistically, we expect backtracking and hesitation episodes to be most frequent in base few-shot traces on human-hard items and to shrink after preference tuning. We also expect families with larger social-desirability shifts to lose more difficulty signal. A negative result on the first check, with flat partial correlations across stages, would be just as informative: it would show that post-training does not cause the curse of knowledge and would direct effort back toward simulators trained on human data.
Workshop authors to connect with
- Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction: Chenguang Wang, Ming Li, Xinyue Zeng, Zhuochun Li, Hong Jiao, Tianyi Zhou, Dawei Zhou
- Psychometric Framework for Comparing Alignment- Conditioned Behavior in Large Language Models: Krisha Arora, Anand Kumar Madasamy, Dishanth Arya
References
- Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction
- Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction
- Psychometric Framework for Comparing Alignment- Conditioned Behavior in Large Language Models
- The Narcissus Hypothesis: Descending to the Rung of Illusion
- Breaking the Assistant Mold: Modeling Behavioral Variation in LLM Based Procedural Character Generation
- Generalized Entropy and Solution Information for Measuring Puzzle Difficulty
- SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction
- PS$^2$: Parameterized Control for Fine-Grained Student Proficiency Simulation
- Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics
- Item Difficulty and Response Time Prediction with Large Language Models: An Empirical Analysis of USMLE Items
- UnibucLLM: Harnessing LLMs for Automated Prediction of Item Difficulty and Response Time for Multiple-Choice Questions
- The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations