Which Part of Machine Reasoning Effort Tracks Human Item Difficulty?

When a teacher or test builder writes a new exam question, nobody yet knows how hard it is. The standard way to find out is to give it to hundreds of students and fit a statistical model, which is slow and expensive. A promising shortcut is to let a reasoning model, meaning a language model that writes out long step-by-step solutions before answering, attempt the question, and then read difficulty off how hard the model had to work. Epi2Diff does this. It splits a reasoning model's solution trace into labeled steps such as planning, computing, checking, and backtracking, then predicts human difficulty from the pattern of those steps. It reports a 24.4% lower regression error than fine-tuned encoders and fine-tuned LLMs. The weakness is that this treats one model's effort, in one prompt context, as a measurement of a human property. A separate literature on measuring LLMs with human instruments shows this kind of measurement is fragile. Answers shift with question order, paraphrase, and earlier conversation, and earlier content keeps pulling later answers even when it is irrelevant (Carryover Effect in LLM Personality Measurement; Persistent Instability in LLM's Personality Measurements).

This note proposes treating machine-derived difficulty the way psychometricians treat any noisy rating. We collect the same effort features from several reasoning models under several deliberate context changes, separate the variance that belongs to the item from the variance that belongs to the model, the context, or their interactions, and predict human difficulty only from the item-level part. The bet is that the human-relevant signal is the "intersubjective" part of machine effort: the burden that different models agree on, and that survives rephrasing and context carryover. Model-specific and context-specific effort is mostly noise or outright difficulty inversion, the case where a model struggles on something humans find easy or breezes through something humans find hard.

Core Claim

Our hypothesis is that difficulty predicted from reasoning traces becomes more accurate and more stable when built only from the component of trace effort that is shared across reasoning models and unchanged under context perturbation. We also hypothesize that the size of the model-by-item disagreement flags the items where machine effort and human difficulty diverge. The second claim is the more novel one. It turns the validity critiques from machine psychometrics, which so far have been diagnostic only, into a working tool for the difficulty estimation pipeline.

The tool is a generalizability study, the standard psychometric method for splitting a score's variance into its sources. For an effort feature $x_{imc}$, such as backtracking count, measured on item $i$ by reasoning model $m$ under context condition $c$, we fit

$$x_{imc} = \mu + \alpha_i + \beta_m + \gamma_c + (\alpha\beta)_{im} + (\alpha\gamma)_{ic} + \varepsilon_{imc}.$$

Here $\alpha_i$ is the item effect we want, $\beta_m$ and $\gamma_c$ are overall model and context shifts, the two interaction terms capture item-specific disagreement across models and across contexts, and $\varepsilon_{imc}$ is residual noise. Each feature then gets a generalizability coefficient, defined as $\sigma^2_\alpha$ divided by $\sigma^2_\alpha$ plus the interaction and residual variances scaled by the number of models and contexts averaged. This coefficient tells us how much of the feature is a property of the item. The predictor uses estimated $\hat\alpha_i$ per feature, weighted or selected by that coefficient. The diagnostic uses the per-item size of $(\alpha\beta)_{im}$.

Relation to Prior Work

Three works supply the method we build on. Epi2Diff provides the step taxonomy and the process features. It uses a single model in a single context, and it does not engage with inversion. Reliable and Efficient Amortized Model-based Evaluation shows that a linear model on frozen Llama-3.1-8B embeddings predicts item difficulty in item response theory (IRT), the family of models that place items and test-takers on a shared difficulty/ability scale. It treats difficulty as an invariant property of the static text, which is exactly the assumption we test. SMART simulates students of different ability levels with a preference-tuned model and fits IRT to their answers. It reaches Pearson $r = 0.67$ on Smarter Balanced, against $0.49$ for a ModernBERT regressor. It is the strongest cold-start competitor in the file. Its simulated students come from one model family, and it validates on about 50 items.

Several papers motivate the proposal by finding evidence of the gap. The carryover paper and PERSIST (Tosato et al.) show that LLM item responses move with prior context, order, and paraphrase. In PERSIST, models above 400B parameters still show standard deviations above 0.3 on a 5-point scale under shuffling, and chain-of-thought increases variability. Neither paper asks whether the same fragility contaminates difficulty estimates. The Personality Illusion and Rethinking Psychometric Evaluation of LLMs add a further warning. In the second, sycophancy self-report correlates with behavior at $r=+0.47$ within a session but $r=-0.07$ across sessions, so shared context can manufacture agreement that looks like validity. We therefore treat same-session context as a facet to vary, not a nuisance to ignore.

Other papers constrain the design through negative results. Natural Language to Logic Translation and Logical Reasoning Abilities of Large Reasoning Models document difficulty inversion. Reading Comprehension Question Difficulty finds that LLMs miss the human burden of long, dense passages. OmniThought shows that trace length is decoupled from cognitive difficulty. Together these mean that no trace feature can be assumed valid. Each feature has to earn its place through cross-model agreement and human predictive value. These results seem to contradict Epi2Diff's positive findings. Our design is meant to reconcile them: Epi2Diff's gains may come from the shared component, while inversion lives in the interactions.

Research Program

The first stage builds the measurement grid. For every item we run four to six open-weight reasoning models from different families and training recipes. Each model sees each item under a small set of context conditions. One is a fresh session. One is a paraphrase of the item that preserves meaning. One embeds the item after a block of earlier items, either all easy or all hard. One is a carryover condition modeled on the seed paper: a third-person narrative earlier in the prompt describes the test as "designed for advanced students" or "a warm-up quiz." Each trace is segmented with the Epi2Diff taxonomy, applied by a fixed labeler model whose agreement with a few hundred human-labeled sentences we report. We then compute the Epi2Diff process features plus solution correctness.

The second stage fits the variance decomposition per feature and reports generalizability coefficients. This is the descriptive result: how much of reasoning effort is item property and how much is model or context artifact. It also yields a carryover estimate, the effect of the "advanced" versus "warm-up" narratives on effort features. That estimate directly extends the personality carryover finding to cognitive performance.

The third stage trains difficulty predictors on human IRT difficulty using the item-level estimates $\hat\alpha_i$. The fourth stage uses per-item model-disagreement scores to flag likely inversion items and tests whether flagged items carry larger human-prediction errors. Flagged items would be routed to human piloting rather than trusted.

Baselines

The naive reference point is the dummy regressor that predicts the mean difficulty. On UPSC2M, a frozen text-embedding MLP improves on it by only 7.1% in RMSE and 9.2% in MAE (UPSC2M). That result tells us how little static text carries on hard multi-subject exams, and any trace-based method must clear it by a wide margin there.

The static-text baseline is the amortized approach of Truong et al.: a ridge regressor on frozen Llama-3.1-8B embeddings of the item text, trained on the same splits. A ModernBERT fine-tuned regressor serves as a second text baseline, since SMART reports $r=0.49$ for it on Smarter Balanced.

The strongest trace-based competitor is Epi2Diff itself, run in its native single-model, fresh-session setting. For fairness, we give it the same reasoning model that performs best individually in our grid and the same downstream regressor family and tuning budget. Its paper reports relative gains (8.0% classification, 24.4% RMSE) but no absolute values in this file, so we re-run it rather than quote it.

The strongest cold-start competitor is SMART on Smarter Balanced, with reported Pearson $r = 0.67$. We compare on the identical 49 items under its evaluation protocol. CodeWorkout is also in SMART's evaluation, but no reported result was found on Lacuna for SMART on CodeWorkout. There we re-run SMART if code is available and otherwise report only our own baselines.

Two ablations isolate the key ingredients. The first is plain averaging of Epi2Diff features across models in the fresh-session condition only. This tests whether multiple models alone explain any gain. The second is the decomposition fitted with models but no context facet. This tests whether context perturbation adds anything beyond model diversity.

Every method gets the same item splits, the same downstream regressor search (ridge, gradient boosting, small MLP, equal hyperparameter trials), and, where traces are involved, a matched total generation budget. Single-model Epi2Diff may spend that budget on repeated samples from one model, so the comparison asks how the budget is spent, not whether more compute helps.

Beating the field means a statistically reliable gain in Pearson correlation and RMSE on held-out items over single-model Epi2Diff and over plain cross-model averaging, by paired bootstrap across items. It also means a clear reduction in how far predictions move between context conditions.

Why not just…?

Why not just average the features over several reasoning models? This is the real threat. A large part of any gain could come from ensembling alone. The plain-averaging ablation exists to measure exactly this. If averaging matches the full method on accuracy, we will say so. The contribution then narrows to two things averaging does not provide. One is the per-feature estimate of how much is item property. The other is the per-item inversion flag, which is useless under averaging because averaging discards the disagreement. The proposal stands on accuracy only if context perturbation and reliability weighting add measurable gains over averaging. If they do not, it stands on the diagnostic.

Why not just run every item in a fresh session and ignore context? Fresh sessions remove carryover by construction, but not paraphrase sensitivity or model idiosyncrasy. PERSIST finds instability under rephrasing and shuffling alone. Fresh-session runs also do not tell a test builder whether an estimate would survive the batched, few-shot prompting people actually use. The context facet lets us measure this. If its variance turns out negligible for difficulty features, unlike for personality items, that is itself a useful negative result and simplifies the pipeline.

Why not just count reasoning tokens? OmniThought shows that trace length and cognitive difficulty come apart. Epi2Diff reports that structured episodes beat unstructured approaches. We nonetheless include token count as a feature inside the decomposition. If it reaches a high generalizability coefficient and carries the predictive weight, the episode taxonomy is unnecessary and we will report that.

Why not just simulate students as SMART does? Simulated students currently give the best reported cold-start number on Smarter Balanced. But they are trained on one family, validated on about 50 items, and open to the persona-mimicry concern raised by the Personality Illusion results. Our method needs no preference training and scales to UPSC2M's 8,973 questions. On Smarter Balanced we compare head to head against $r=0.67$. We also test whether SMART's predictions err on the same items our disagreement score flags, which would indicate a shared machine-human gap rather than method noise.

Proposed Experiments

The data are the four human difficulty datasets used by Epi2Diff, Smarter Balanced and CodeWorkout from SMART, and UPSC2M, whose 1.96 million attempts allow stable IRT difficulty for a large, multi-subject item pool. Targets are IRT difficulties fitted on the human response data. Metrics are Pearson and Spearman correlation and RMSE against human difficulty, plus prediction shift between context conditions. For the diagnostic, the metric is the area under the ROC curve for the disagreement score in identifying the top decile of absolute human-prediction residuals under the static-text baseline.

Four checks decide the outcome. First, the item-level component must beat single-model Epi2Diff and plain averaging on held-out items in at least most datasets. Second, the carryover narratives must produce a measurable shift in effort features, and a narrative-matched shift in predicted difficulty, under single-model prediction but a reduced shift under the decomposed predictor. Third, the model-disagreement score must flag large-error items better than chance and better than a trace-length heuristic. Fourth, as a falsification test, a hand-built set of inversion probes adapted from the logic-translation and syllogism studies should receive high disagreement scores. If they receive low disagreement, inversion is shared across reasoning models, and the premise that cross-model agreement equals human relevance is wrong.

Any of these outcomes is informative. Positive results give test builders a cheaper and more robust cold-start estimate with a built-in warning light. Negative results tell the field that LLM effort is not the intersubjective signal it has been treated as.

Workshop authors to connect with