Steered Respondent Populations for Estimating Item Discrimination

Built from these papers presented at Social Sim @ COLM 2026:

Test designers care about two things for each question. One is how hard it is. The other is how well it separates stronger students from weaker ones, which psychometricians call discrimination. Language models can now give a rough estimate of difficulty before any student sees an item. They are poor at estimating discrimination. Across 42 models, the best direct guess correlates with human-calibrated discrimination at only Spearman 0.152. Answers collected from a pool of prompted "student personas" do somewhat better, reaching 0.241 (Chen et al., Seed Paper 2).

Our claim is that discrimination is a property of a population, not of a single answer. An item discriminates when the people who get it right also tend to score well overall. A synthetic crowd can only reproduce that if two conditions hold: its members vary along a shared ability axis, and they also vary in other ways, such as carelessness or anxiety, that cause errors unrelated to ability. We propose to build such crowds inside one open model. We do this with activation steering, which nudges the model's hidden states along learned directions at inference time. One direction carries reading ability; the others carry Big Five personality traits. We then fit the distribution of steering settings so that the synthetic crowd matches human statistics on a small set of anchor items with known parameters. Finally, we read off discrimination for new items. The new ingredient is this population-level fit in steering space. The test of whether it matters is whether it beats the same fit applied to prompted personas.

Core Claim

Item response theory (IRT) models the chance that respondent $j$ answers item $i$ correctly. In its two-parameter form this is $\sigma(\alpha_i(\theta_j - \beta_i))$, where $\theta_j$ is the respondent's ability, $\beta_i$ the item's difficulty, $\alpha_i$ its discrimination, and $\sigma$ the logistic function. The multidimensional version replaces scalar ability with a vector (Learning Compact Representations of LLM Abilities via Item Response Theory).

Estimating $\alpha_i$ from synthetic respondents fails for two reasons. Prompted personas collapse into extreme caricatures with unrealistically clean fit (Designing AI-Agents with Personalities: A Psychometric Approach). And a single model's respondents cluster in a narrow ability band (Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis). Both failures distort the covariance structure that discrimination depends on.

We hypothesize that a crowd with continuous, independently controllable variation fixes both, provided its population weights are fit to human anchor data rather than chosen by hand. The variation needed is in ability, plus personality traits that add structured, ability-independent error.

The personality component is a substantive claim, not decoration. Items that invite careless slips should have lower human discrimination. Only a crowd that varies in conscientiousness-like traits can show this. That is why the proposal is multidimensional rather than another scalar-ability simulator.

Relation to Prior Work

The closest work is SMART. It uses preference optimization to train Llama-3.2 to answer like a student at a given scalar ability, then fits an IRT model to the synthetic answers. It reaches Pearson 0.67 for difficulty on Smarter Balanced reading. SMART differs from our proposal in three ways. Its respondents vary along one axis. It estimates only difficulty. Its authors name multidimensional personas as missing. We keep SMART's simulate-then-fit logic, but target discrimination, use a multi-trait steering space, and fit the population mixture rather than training the model.

Seed Paper 2 supplies both the target and the baseline. It shows that prompted personas carry some discrimination signal, but weakly. It does not ask whether the weakness comes from the persona mechanism or from an uncalibrated population mix. Our design separates these two causes.

Leveraging LLM-Respondents shows that ensembles of different foundation models widen the ability spread and reach Spearman 0.93 for difficulty on 20 algebra items. This motivates our concern with spread, but it does not address discrimination.

Do LLMs Give Psychometrically Plausible Responses in Educational Assessments? constrains the design. Temperature scaling lowers the mismatch with human answer distributions but leaves models blind to plausible distractors. So we cannot expect logit softening to manufacture human-like errors.

The steering machinery comes from two sources. PERSONA extracts Big Five directions by contrasting activations under opposite prompts and reports them as roughly orthogonal and composable. Activation-Steered Personas applies trait-pole vectors with scalar strengths to Qwen2.5-7B and Llama-3.1-8B. That paper also contains a warning we take seriously: socially expressive traits become observable under steering, while traits like "consistent" stay hard to detect in short interactions.

A further caution comes from reports that alignment resists steering toward low-conscientiousness traits (PERSONA; Breaking the Assistant Mold). Steering may therefore change capability non-monotonically. We measure this effect rather than assume it away.

Research Program

Building the steering grid. For each base model (Qwen2.5-7B-Instruct, plus Llama-3.1-8B-Instruct for robustness), we extract one ability direction and five trait directions.

The ability direction contrasts activations on the same passages under two framings: answers from strong readers versus weak readers. As a check, we also extract it a second way, by contrasting activations on correct versus incorrect model answers. We verify that steering along it lowers accuracy smoothly and monotonically on held-out items, without breaking fluency.

The trait directions follow the contrastive recipe from PERSONA.

We then lay out a grid $G$ of steering-coefficient vectors $c_g$. Ability takes 9 levels. Each trait takes 3 levels, and the grid is sampled with a Latin hypercube design to keep it to about 400 cells. For every cell and every item, we record the probability $p_g(i)$ of a correct response. For multiple-choice items this comes from option log-probabilities. For open-ended items it comes from sampled answers scored by the same automated-grader approach SMART uses.

Fitting the population. A synthetic population is a weight vector $w$ over grid cells. We fit $w$ on anchor items whose human difficulty and discrimination are known:

$$ \hat w = \arg\min_{w \in \Delta} \sum_{i \in A} \big(\hat\rho_i(w) - \rho_i\big)^2 + \lambda \big(\hat\pi_i(w) - \pi_i\big)^2 $$

The symbols are as follows:

  • $A$ is the anchor set.
  • $\Delta$ is the probability simplex, so the weights are non-negative and sum to one.
  • $\rho_i$ is the human item-total (point-biserial) correlation.
  • $\pi_i$ is the human proportion correct.
  • $\hat\rho_i(w)$ and $\hat\pi_i(w)$ are the same statistics computed on respondents drawn from the mixture, where each respondent answers item $i$ correctly with probability $p_g(i)$.
  • $\lambda$ trades off the two terms.

Because the $p_g(i)$ are precomputed, fitting the population needs no further model calls. We optimize with Monte Carlo estimates, using a fixed random seed so the objective is smooth enough for projected gradient descent.

For held-out items, we sample a population from $\hat w$, fit a two-parameter IRT model, and report its $\alpha_i$. We also report the classical item-total correlation, so results are comparable with Seed Paper 2's classical-test-theory numbers.

Analysis. We read the fitted $\hat w$ as a statement about the population. Where does it place mass along the ability axis? Does it spread mass along conscientiousness and neuroticism?

We test the mechanistic hypothesis directly. Annotators label held-out items for slip-proneness: negations, long distractors that closely match passage wording, and multi-part answers. We then check whether the trait dimensions specifically improve discrimination prediction on those items.

Finally, we plot item response curves as a function of the steered ability coefficient. This locates any region where trait steering causes capability collapse rather than graded error.

Data

The primary benchmark is the reading-comprehension discrimination set from Seed Paper 2. Its human-calibrated discrimination values and its 0.241 baseline are the target.

The second benchmark is Smarter Balanced grade-6 reading, used in SMART. It has about 85,000 human responses across 49 items, from which human two-parameter and partial-credit discrimination can be computed. With only 49 items, we use leave-k-out folds rather than one split.

The third is the OpenStax algebra set from Leveraging LLM-Respondents. It has 20 items and 50 human respondents. We use it only as a small out-of-domain check, since 50 humans give noisy discrimination estimates.

We considered the NYSTP/STAAR items in Prediction of Item Difficulty. They are useful if discrimination parameters can be obtained, but the file reports only difficulty for them.

Baselines

Text-only prediction. The naive reference is direct zero-shot discrimination prediction from item text. The best reported value is Spearman 0.152 (Chen et al., Seed Paper 2). Alongside it, we train a supervised regressor on sentence embeddings of the item text, using the same anchor items as training data. Text regression of discrimination reached only test $R^2$ = 0.13 on HotpotQA in Predicting Difficulty and Discrimination of Natural Language Questions. That was a different setting, with model-based rather than human respondents, so we rerun it on our data. Any useful method must clearly beat both versions of this baseline.

Prompted personas, as reported. This is the strongest published result on the question: the all-persona prompted respondent pool, which reached Spearman 0.241 with human discrimination (Chen et al., Seed Paper 2). We reproduce it with the same base models we steer, so model identity is not a confound. We report both their number and our reproduction.

Prompted personas with the same anchor fit. This is the fairness-critical version of the previous baseline. We build a prompted grid of the same size: ability-level and Big Five descriptions written in the expanded natural-language style that Designing AI-Agents with Personalities found most faithful. We give it the identical anchor-fitting procedure, the same number of grid cells, the same generation budget, and the same anchors. Beating this baseline is the evidence that steering, and not the anchor fit, carries the gain.

SMART-style difficulty-conditioned simulation. We include a scalar-ability simulator, conditioned on ability as in SMART. Where training is affordable we use SMART's preference-optimization approach; otherwise we approximate it with an ability-prompt grid. Its discrimination is computed by the same downstream procedure. No reported result was found on Lacuna for SMART's discrimination accuracy; its reported figure is difficulty Pearson 0.67 on Smarter Balanced.

Ablations. Two ablations isolate the key ingredient. Ability-only steering keeps the ability direction but drops the trait directions; this tests whether personality dimensions add anything. Random-direction steering replaces the trait directions with random directions of matched norm; this tests whether any extra variance would do.

Success criterion. We count the proposal as working if two conditions hold. First, the steered, anchor-fit population beats the anchor-fit prompted grid on held-out Spearman with human discrimination, with a bootstrap confidence interval over items that excludes zero. Second, it beats ability-only steering, specifically on the slip-prone item subset.

Why not just…?

Why not just prompt richer personas and sample at temperature? This is the approach Seed Paper 2 already tested, at 0.241. Prompting has two documented problems: binary or terse trait prompts produce caricatured, overly clean populations (Designing AI-Agents with Personalities), and temperature does not create human-like distractor choices (Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?). Still, it is possible that most of the gap comes from the hand-chosen population mix rather than from prompting itself. That is why the anchor-fit prompted grid is a full competitor. If it matches steering, we conclude the contribution is the population fit, and we report that honestly.

Why not just ensemble different foundation models? Ensembles widen ability coverage and reached Spearman 0.93 for difficulty (Leveraging LLM-Respondents). But no reported result was found on Lacuna for ensemble-based discrimination. Ensembles also vary corpus, size, and alignment all at once, so they cannot show whether non-ability traits drive the low-discrimination items. We include an anchor-fit ensemble of open models, matched in respondent count, as an additional comparison. If it wins, the steering-space argument is weakened to a cost argument.

Why not just regress discrimination from item text? Text-feature regression explains little variance in discrimination (Predicting Difficulty and Discrimination of Natural Language Questions). For difficulty, direct prediction trailed simulation, at 0.49 versus 0.67 in SMART. Because our method uses labeled anchors, the fair comparison gives the regressor the same anchors. We report learning curves over anchor counts for both approaches.

Why not just steer ability alone? It is simpler, and if personality adds nothing it is the better method. The ability-only ablation answers this directly. The slip-prone item analysis says where any difference should show up.

Decisive Checks

Sanity check on the ability direction. Before anything else, steering along the ability direction must produce monotone item response curves on held-out items. If it does not, the population is not organized by ability, and the item-total statistics are meaningless.

Main comparison. The decisive test compares the steered and prompted grids under identical anchor fitting. We report held-out Spearman with human discrimination on the Seed Paper 2 items and on Smarter Balanced folds.

Anchor-count curve. We plot accuracy against the number of anchor items, at 5, 10, 20, and 40. This shows how much human data the method actually saves.

Prediction about traits. The trait hypothesis makes a falsifiable prediction. Mass in $\hat w$ should spread along conscientiousness and neuroticism, and removing those directions should hurt mainly on slip-prone items. If random directions do as well, the "personality" interpretation fails, and we report the method as variance injection.

Alignment check. We report how the steered low-conscientiousness cells behave. If they produce collapse rather than graded slips, that confirms the alignment restoring force reported in prior work and bounds the usable region of the steering space.

Workshop authors to connect with