Advancement or Acceleration? How Data Curation Improves LLM Training

Jiaxing Wang, Zicheng Zhang, Deping Xiang, Jin Xu, Yong Li, Chao Liu, Pengzhang Liu, Qixia JiangarXiv 2026 · arXiv

TL;DR

Does data curation actually make models better, or just faster to train? This paper formalizes the distinction between advancement (improving the final performance ceiling) and acceleration (reaching a performance level with less data). The authors demonstrate that curation only provides lasting "advancement" when it corrects a mismatch between training and target distributions, and they introduce a simple, low-cost $\Delta$-test to predict these gains before starting expensive training runs.

What Problem Does This Paper Address?

In the race to build better Large Language Models (LLMs), researchers have moved beyond simple scaling to sophisticated data curation—strategies for picking the "best" data mixtures or individual examples. However, the field is plagued by contradictory results: some papers report massive gains from curation, while others find that simple uniform mixtures or random sampling are surprisingly hard to beat at scale.

The authors argue that this confusion stems from conflating two different effects. If a curation method shows a 5% gain halfway through training, is that because the model is fundamentally better, or because it's just learning the same thing faster? This paper seeks to provide a theoretical and empirical framework to separate these two phenomena and identify the conditions under which curation actually "advances" a model's capabilities.

Methods

The researchers approach the problem through three lenses:

  1. Analytical Theory: They use a "teacher-student" framework (specifically perceptron learning) to derive exact equations for how models behave under different data allocations. This allows them to mathematically define the "endpoint" of training.
  2. The $\Delta$-test: They propose a lightweight diagnostic tool. By comparing a baseline model's in-sample training loss with its loss on a target evaluation set, they calculate $\Delta$. A high $\Delta$ indicates a "correctable gap" where curation has the potential to provide advancement.
  3. LLM Experiments: They validate their theory by training GPT-style models (160M and 1.1B parameters) across various curation scenarios, including data mixture (balancing domains) and data selection (picking specific examples).
The theoretical setup for data mixture and selection, showing how "gaps" are created.

Datasets

  • Data Mixture: Experiments used a 6-billion token subset of SlimPajama, covering seven domains: ArXiv, Books, C4, CommonCrawl, GitHub, StackExchange, and Wikipedia.
  • Data Selection: The candidate pool was a 5-billion token subset of OpenWebMath.
  • Evaluation Targets: For selection, they used MetaMath and Mammoth to create "shifted" targets that represent a specific distribution gap compared to the general web data.

Key Insights and Ideas

  • Advancement is a Property of the Task, Not Just the Method: Curation cannot "invent" performance. It can only provide advancement if there is a "train-target gap"—a mismatch between what the baseline data provides and what the evaluation requires. Without this gap, even the most sophisticated curation method only provides transient acceleration.
  • The "Aha" Moment on Acceleration: Many reported gains in curation literature are actually just acceleration. As the training budget grows (more tokens per parameter), the advantage of picking "hard" or "informative" examples often vanishes, and the baseline eventually catches up.
  • Predictive Power of $\Delta$: The $\Delta$-test is highly predictive. If $\Delta \approx 0$, you are better off spending your compute on scaling (more data/parameters) rather than complex curation. If $\Delta$ is large, curation is likely to yield persistent, non-vanishing gains.
Left: In a no-gap setting, curation gains vanish over time (acceleration). Right: Final gains correlate strongly with the initial $\Delta$ value.

Key Results

  • Mixture Gains: In a "no-gap" setting where the training pool matches the target, optimized mixtures (like RegMix or TANDEM) showed early speed-ups but converged to the same performance as a simple uniform baseline. However, when a gap was introduced (e.g., holding out CommonCrawl), curation reduced perplexity from 49.19 to ~40.2, a gain that persisted until the end of training.
  • Selection Gains: For data selection, target-aware methods (like RHO-1 and BLADE) only outperformed random sampling when the target distribution was shifted. In "in-distribution" settings, aggressive pruning actually hurt final performance.
  • Scale Invariance: The diagnostic power of the $\Delta$-test held true as models scaled from 160M to 1.1B parameters, suggesting it can be used on small "proxy" models to plan large-scale training runs.
Comparison of different curation methods across no-gap and shifted-gap scenarios.

Limitations and Future Work

  • Model Scale: While the authors tested up to 1.1B parameters, the behavior of curation at the 70B+ scale remains an open question.
  • Static Limits: The theoretical model relies on high-dimensional limits and doesn't fully capture the complex, non-linear training dynamics of modern transformers or the impact of specific data ordering (curriculums).
  • Target Construction: The paper suggests that "target design"—carefully defining what we want the model to be good at—is perhaps more important than the specific algorithm used to select the data. This points toward a new research direction focused on high-fidelity target set construction.

Summary updated

Abstract

While data curation is widely credited with improving large language model training, a basic question remains unresolved: does curation expand what a model can ultimately learn, or merely reach the same solution faster? In this paper, we formalize this distinction between advancement (improving the attainable performance) and acceleration (reaching the same performance with less data). Through an analytic characterization and real-world LLM experiments, we show that advancement requires a correctable train-target gap. When the baseline training and target objectives coincide, no curation policy improves the endpoint, although curation may still accelerate training. This reframes advancement as a property of the curation task rather than the curation method. Under a common gap-centric view of data mixture and selection, we operationalize this condition as the $Δ$-test, an observable discrepancy between the uncurated baseline's training and target losses that is computable prior to expensive curation. Across extensive experiments, an appreciably nonzero $Δ$ predicts persistent gains, whereas gains near $Δ=0$ tend to diminish as the training budget grows. Beyond individual methods, our findings recenter data curation on the task itself and point to target-set construction as a promising direction for shaping what curation can achieve.
BibTeX
@misc{wang2026advancement,
  author = {Jiaxing Wang and Zicheng Zhang and Deping Xiang and Jin Xu and Yong Li and Chao Liu and Pengzhang Liu and Qixia Jiang},
  title = {{Advancement or Acceleration? How Data Curation Improves LLM Training}},
  year = {2026},
  eprint = {2610.36983},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36983},
}
Figure