When we ask which training data gave a language model a skill such as social reasoning, current attribution methods answer a narrower question. They ask which training documents lowered the model's loss on a benchmark, with every test question counted equally. Benchmarks are not uniform, though. Many questions are answered correctly by almost every model, some are answered correctly mostly by weak models because they are mis-keyed, and only a fraction separate strong models from weak ones. Item response theory (IRT), the statistical model behind standardized testing, makes this precise. It gives each question a difficulty and a "discrimination", which measures how sharply the chance of a correct answer rises with the test-taker's underlying ability. IRT then estimates that ability as a single hidden number, θ. This proposal derives the influence of a training example on θ itself. The result turns out to be simple: an influence score per test question, weighted by that question's discrimination and pooled into a single query vector. That is cheap enough to run at pretraining scale with existing compressed-gradient tools.
The seed provenance audit makes the stakes concrete. Tracing LLM Social Reasoning to Pretraining Data traces OLMo3-7B's SocialIQA behavior to a diffuse spread of narrative and interpersonal web regions, but it averages over all SocialIQA questions. Social-commonsense items are known to carry annotation artifacts (SocialIQA), and Evaluation Examples are not Equally Informative shows that negatively discriminating items often signal mis-keying. The diffuse profile could therefore reflect how social reasoning is learned, or it could come from averaging over uninformative and broken items. Attributing to ability rather than loss separates these two explanations, and it predicts which data matters for questions the attribution never saw.
Core Claim
Fix a two-parameter IRT model with item discriminations $\alpha_i$ and difficulties $\beta_i$, calibrated on a population of other models. Let $q_i(w)$ be the evaluated model's probability of the correct option on item $i$ given weights $w$. The model's soft ability estimate $\hat\theta(w)$ solves the maximum-likelihood score equation
$$\sum_i \alpha_i \big(q_i(w) - P_i(\hat\theta)\big) = 0, \qquad P_i(\theta) = \sigma\big(\alpha_i(\theta - \beta_i)\big).$$
Upweight a training example $z$ by $\epsilon$ and differentiate implicitly. The influence of $z$ on ability is
$$\frac{d\hat\theta}{d\epsilon} = \frac{\sum_i \alpha_i \, \mathcal{I}(z, q_i)}{\sum_i \alpha_i^2 P_i(1-P_i)},$$
where $\mathcal{I}(z, q_i)$ is any standard influence estimate of $z$ on $q_i$, and the denominator is the test information at $\hat\theta$, which is constant across training examples. Three consequences follow.
First, ranking training data by its effect on ability is the same as ranking it by a discrimination-weighted sum of per-item influences. Mis-keyed items, with $\alpha_i<0$, enter with the opposite sign, and items with flat response curves drop out.
Second, because influence is linear in the test gradient, the weighted sum collapses to one query vector $\sum_i \alpha_i \nabla q_i$. The whole benchmark therefore costs one inverse-curvature product. Most items contribute little, so the sum can be truncated to the items with the highest Fisher information at $\hat\theta$, a few dozen to a hundred. The error bound comes from the dropped terms' share of the information.
Third, θ is defined to generalize across items. The empirical claim is therefore that ability attribution computed on one set of items predicts the counterfactual effect of removing data on a disjoint held-out set of items better than loss attribution does. That claim is testable and could fail.
This differs from the closest work. Which Data Attributes Stimulate Math and Code Reasoning? uses influence functions with heuristic difficulty labels and never estimates ability or discrimination. Dynamic Data Selection for Curriculum Learning via Ability Estimation uses IRT ability to schedule training data but never computes counterfactual influence. DUET combines influence functions with scalar downstream feedback rather than a calibrated latent trait. None of them differentiates an IRT ability estimate with respect to training data.
Research Program
Item calibration. Calibrate 2PL item parameters for MMLU, ARC-Challenge and HellaSwag from public leaderboard response matrices. These are the item banks used by tinyBenchmarks and Adaptive Testing for LLM Evaluation. SocialIQA is not covered by those matrices, so we collect responses from roughly a hundred open models. Discrimination must come from real responses: the seed paper LLMs Struggle to Measure What Distinguishes Students shows that prompting models for discrimination reaches only Spearman 0.152, and synthetic respondent pools reach 0.241. This rules out the shortcut. We fit a 3PL model with a guessing floor as a robustness check, since all four benchmarks are multiple-choice. The derivation carries over with modified weights.
Influence backend. The influence estimate $\mathcal{I}(z, q_i)$ is treated as interchangeable. We use TrackStar through Bergson, which matches the seed audit, and a TRAK-style random projection. Unlearning Traces the Influential Training Data (UnTrac-Inv), which unlearns the test set once, serves as a check that does not rely on second-order approximations. Its "test set" becomes the discrimination-weighted objective. Following First is Not Really Better Than Last, gradients are taken from middle-layer value projections.
Aggregation. Influence is aggregated over the 576 WebOrganizer format-by-topic bins of Dolma3, as in the seed audit. This lets us directly re-run the seed audit's comparison of SocialIQA, MMLU Social Sciences, ARC-Challenge and MMLU STEM, with loss attribution replaced by ability attribution.
Ground truth. Ground truth for "which data matters" comes from intervention rather than from another attribution score. For a one-billion-parameter OLMo-family model, we run continued pretraining from a mid-training checkpoint on mixtures with selected bins removed, then measure the actual change in θ and in accuracy on held-out items.
Baselines
Do nothing. The naive reference point ranks bins by their share of the training mixture, which is the prediction if influence were proportional to exposure. Any attribution method that cannot beat corpus share on held-out bin-removal effects is not informative.
Uniform loss attribution. This is the strongest existing method and the one the seed audit uses. It is TrackStar attribution to the mean loss over all benchmark items, run with the same projection dimension, the same layers, the same training-data working set, and the same compute for the training-side gradients. Its only differences from our method are the item weights and the item subset. We found no numerical attribution-accuracy figure on Lacuna for the seed paper's method: it reports qualitative profiles and unlearning effects. For the curated-subset idea, the relevant number is from tinyBenchmarks: 100 IRT-chosen items track full-benchmark accuracy within 1.9% average error with Spearman rank correlation often above 0.95. That figure concerns evaluation, not attribution, which is exactly the gap tested here. We found no reported result on Lacuna for UnTrac's numerical correlation with leave-out retraining on these benchmarks. Beating the strongest baseline means a higher Spearman correlation between predicted and actual held-out bin-removal effects, with non-overlapping bootstrap intervals over bins, at equal or lower test-gradient count.
Uniform weights over the same high-information subset. This ablation removes the discrimination weights while keeping the item subset, separating "fewer items" from "weighted items."
Single-model weighting. This is the key-ingredient ablation, described in the next section.
Why not just…?
Why not just sum per-item influences uniformly over the full benchmark? Because the derivation shows this is ability attribution with every discrimination forced to one. It lets saturated items add noise and lets mis-keyed items push in the wrong direction. Whether this matters in practice is the main experiment: if uniform and discrimination-weighted attribution predict held-out removal effects equally well, the proposal's central claim fails, and we will report it that way.
Why not just weight items by how uncertain this one model is, without calibrating on a population? This is the most serious threat. The gradient of $q_i$ already contains a $q_i(1-q_i)$ factor, so saturated items are partly downweighted without any IRT. Population calibration adds two things this cannot: the sign of discrimination, which flags mis-keyed items, and a slope that reflects how items separate other models. We run the uncertainty-only weighting as a named ablation. If it matches the full method, we will narrow the contribution to the claim that negative-discrimination filtering plus truncation is all that matters, rather than argue for the 2PL machinery.
Why not just drop negatively discriminating items and average the rest? This captures the mis-keying benefit at nearly zero cost. Evaluation Examples are not Equally Informative shows such items exist. Our derivation predicts that the graded positive weights still matter, because low-but-positive discrimination items dominate social benchmarks in count. The ablation settles it.
Why not just prompt a frontier model for item quality, or use length or perplexity? The file's evidence rules these out as weight sources. Direct discrimination prediction reaches Spearman 0.152 in the seed study, and DDaCLAE reports nearly zero correlation between surface heuristics and IRT difficulty. We include the perplexity weighting only as a cheap baseline.
Proposed Experiments
Held-out prediction test. Each benchmark's items are split in half. Attribution is computed on half A under each weighting scheme. Then 40 bins are chosen by stratifying across the attribution rankings, and each is removed in a separate continued-pretraining run, along with a dozen random multi-bin removals. On half B we measure the change in θ (re-estimated with the fixed item parameters) and the change in raw accuracy. The decisive check is whether ability attribution's Spearman correlation with actual held-out change exceeds that of uniform loss attribution and the single-model uncertainty ablation. The check is run separately for SocialIQA, where we expect discrimination to matter most, and for MMLU STEM, where items are cleaner and the gap should shrink. A gap appearing on SocialIQA but not STEM would be the expected pattern; no gap on either would falsify the proposal.
Seed-audit re-run. We repeat the seed audit on OLMo3-7B with ability attribution and report whether SocialIQA remains the diffuse outlier. Diffuseness is measured as the entropy of positive influence over the 576 bins, compared against the other three tasks. If the outlier status disappears when negative and near-zero discrimination items are removed from the weighting, the seed paper's headline finding is partly an item-quality artifact. If it survives, the finding is strengthened. We then run influence-targeted unlearning on the top bins under each ranking, as the seed paper did, and compare the resulting drop in held-out θ.
Cost curve. We plot held-out predictive correlation against the number of test items whose gradients are computed. The comparison is between Fisher-information truncation at $\hat\theta$, random subsets, and tinyBenchmarks anchor points, from 10 items up to the full benchmark. The scalability claim holds if the information-truncated curve reaches its plateau with an order of magnitude fewer test gradients than random subsets.
Robustness to calibration population. We recalibrate item parameters on leaderboard models excluding the OLMo family. This addresses the concern from Adaptive Testing for LLM Evaluation that parameters fit to one model cohort may not transfer to another. The attribution rankings should remain stable under this change.
Workshop authors to connect with
- LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment: Han Chen, Ming Li, Chenguang Wang, Yijun Liang, Dawei Zhou, Hong Jiao, Tianyi Zhou
- Tracing LLM Social Reasoning to Pretraining Data: A Provenance Audit for Social Simulation: Glenn Matlin, Chandreyi Chakraborty, Saehee Eom, Mika Okamoto, Rayan Castilla, Louis Jaburi, Alvin Deng, Taywon Min, Lucia Quirke, Stella Biderman, Mark Riedl
References
- Tracing LLM Social Reasoning to Pretraining Data: A Provenance Audit for Social Simulation
- SocialIQA: Commonsense Reasoning about Social Interactions
- Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
- Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions
- Dynamic Data Selection for Curriculum Learning via Ability Estimation
- DUET: Optimizing LLM Training Data Mixtures via Noisy Feedback from Unseen, Downstream Evaluation Tasks
- tinyBenchmarks: evaluating LLMs with fewer examples
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
- LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
- TRAK: Attributing Model Behavior at Scale
- Unlearning Traces the Influential Training Data of Language Models
- First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation