Data Attribution Should Steer Pretraining

Glenn Matlin, Chandreyi Chakraborty, Mohammed Rehan Parwani, Saehee Eom, Anthony Wen-Ming Zang, Akhil Theerthala, Arjun Chatterjee, Tejas Khandwekar, Brandon Michaels, Alvin Deng, Mark RiedlarXiv 2026 · arXiv

TL;DR

This position paper argues that training-data attribution (TDA) should move beyond post-hoc analysis and one-off document rankings. Instead, researchers should use TDA to identify testable, reusable data properties that can steer future pretraining runs more efficiently than standard baselines like perplexity filtering, even when accounting for the "discovery cost" of the attribution itself.

What Problem Does This Paper Address?

In the current LLM landscape, "data is king," but we are surprisingly bad at explaining why specific data is valuable. Developers use various heuristics—like perplexity filtering, quality classifiers, or domain mixing—to curate pretraining sets. While Training Data Attribution (TDA) tools exist to identify which documents influenced a model's behavior, they are typically used for "historical" analysis: "Which documents made this specific model toxic/smart?"

The authors identify a major gap: influence rankings are usually "consumed" by the run they were computed for. They don't easily translate into generalizable rules for the next model. Furthermore, what makes a document valuable changes as a model evolves; data that helps a "toddler" model learn basic syntax might be redundant for an "adult" model. The paper seeks to turn TDA into a prospective steering tool—a way to discover "capability provenance"—connecting specific model abilities to the data properties and training conditions that shaped them.

Methods

The paper proposes a conceptual and procedural framework for "Capability Provenance" as a control loop:

  1. The Attribution-to-Steering Control Loop: Instead of just ranking documents, developers should observe learning, attribute it to sources, interpret those sources to find a property (e.g., "high token density of specific logic keywords"), and then test that property as a new curation criterion in a bounded trial.
  2. The Ladder of Information: The authors define five levels of evidentiary claims, ranging from simple historical contribution to understanding the underlying learned mechanism. Crucially, they specify what each level cannot establish (e.g., knowing which data contributed historically doesn't prove that data will be useful in a future training stage).
  3. Matched Total Cost Evaluation: To make TDA a rigorous engineering tool, the authors propose a "falsifiable comparison." Any new curation rule derived from expensive TDA must be evaluated against "cheaper" selectors (like perplexity) at a matched total compute budget. This means the cost of the TDA "discovery" phase must be charged against the TDA-derived method.

Datasets

As a position paper, this work does not introduce a new dataset but rather synthesizes findings from several major pretraining corpora and benchmarks, including:

  • English C4 and The Pile: Used to discuss how quality signals diverge from actual usefulness.
  • RedPajama-V2: Used to highlight disagreements between static quality classifiers and token utility.
  • OpenWebText: Referenced in a case study on tracing toxicity in the Pythia-160M model.
  • DCLM-Baseline and Dolma: Proposed as the standard testbeds for their "matched-cost" research agenda.

Key Insights and Ideas

  • Historical Contribution $\neq$ Future Value: Data that was essential for forming a capability early on may become redundant later. Conversely, some data only becomes "learnable" once certain prerequisites are met. TDA should help us understand these stage-dependent shifts.
  • Explanation is the Distinctive Utility: The real power of attribution isn't the ranking itself, but the explanation it provides. If we can identify a structural or lexical property that explains a gain, we can build a cheap "selector" (a classifier or filter) that doesn't require running expensive attribution ever again.
  • The Granularity Mismatch: The authors highlight a "published example" where document-level attribution failed to reduce toxicity without destroying model fluency. However, by moving to a finer granularity (token-density scoring), researchers were able to cut toxicity by 4–10x. This suggests that the "unit of attribution" must match the "unit of intervention."

Key Results

The paper is primarily theoretical and methodological, but it provides a rigorous Research Agenda (Section 6) and an Illustrative Reference Protocol (Table 2).

The authors argue that the field should adopt a "seeds and decision rule" where an attribution-derived criterion is only considered successful if it beats the best compute-matched baseline (like a perplexity-based selector) across at least three independent pretraining seeds, while staying within pre-declared "guardrail" metrics (like MMLU or HellaSwag).

They re-analyze the work of Coalson et al. (2025) to show how their "Ladder of Information" clarifies what specific experiments actually license for curation. For instance, they show that while document deletion failed, a more targeted token-level suppression successfully transferred across model scales.

Limitations and Future Work

  • Estimator Error vs. Hypothesis Error: A failed run might mean your theory about the data was wrong, or it might just mean your TDA tool (the "estimator") was noisy. The authors suggest using small-model retraining to isolate these errors.
  • Unlearning as a Proxy: Many TDA studies use "unlearning" (removing data) to prove influence. The authors warn that this is an imperfect proxy because models can often recover suppressed behaviors from other, independent facts in the dataset.
  • Dual-Use Risks: "Capability provenance" is a double-edged sword. While it helps with safety audits and bias reduction, it could also be used to guide targeted poisoning or to concentrate harmful skills in a model. The authors emphasize that empirical contribution should never be seen as "permission" to train on contested or harmful data.

Summary updated

Abstract

Pretraining data curation decides what a model learns from, yet what makes a document worth learning from depends on what the model already knows. Influence-driven selectors already rank documents to improve one run; we argue that training-data attribution should steer pretraining by identifying testable data properties that become reusable curation criteria, evaluated on new data against cheaper selectors at matched total cost, discovery included. A document ranking is consumed by the run it was computed for, whereas an explained data property can be tested across corpora and checkpoints, especially where familiar quality signals diverge from measured usefulness. Because historical contribution, an understandable explanation, and future data value require distinct evidence, we specify a feedback loop that interprets attributed sources, tests candidate data changes, and states for five claim types what evidence each requires and what none establishes. We close with a falsifiable comparison against perplexity, classifier, and mixing baselines at equal total compute.
BibTeX
@misc{matlin2026data,
  author = {Glenn Matlin and Chandreyi Chakraborty and Mohammed Rehan Parwani and Saehee Eom and Anthony Wen-Ming Zang and Akhil Theerthala and Arjun Chatterjee and Tejas Khandwekar and Brandon Michaels and Alvin Deng and Mark Riedl},
  title = {{Data Attribution Should Steer Pretraining}},
  year = {2026},
  eprint = {2611.19698},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19698},
}
Figure