Measuring When a Language Model Can Absorb Each Kind of Data

Farmers learned long ago that putting all the fertilizer down at sowing wastes most of it. Early in the season the crop cannot take it up, so the nutrients wash away, and the soil runs short later when demand peaks. Each nutrient also peaks at a different growth stage. Language-model pretraining has a close analogue. Most runs feed the model one fixed mixture of data sources (web text, code, encyclopedias, papers) from start to finish. Recent work shows that changing the mixture over time helps, but every existing method decides when to feed a source by looking at a single loss number or at a benchmark score measured once, at the end.

This proposal measures directly how much each skill of a model gains from each data source at each stage of training. The method is to fork training at checkpoints and apply short, randomized "pulses" of one source. Within this note, a pulse means replacing part of the mixture with one data source for a short stretch of tokens, while a sibling copy of the model continues on the unchanged mixture. The result is a stage-by-source-by-skill map of uptake: how much a given skill improves per token of a given source at a given point in training. We then schedule sources to match that map. Because the pulses are real interventions and follow a clear time order, the map avoids a problem that affects both end-of-run benchmarking and observational epidemiology: when two things are measured at the same moment, you cannot tell which one caused the other.

Core Claim

Our claim is that the value of a data source to a specific skill changes with training stage, and that it changes differently for different skills. We also claim that this stage dependence can be measured cheaply enough, with forked pulse experiments on small models, to build a schedule that beats both loss-driven dynamic mixing and a simple two-stage end-of-training upsample. The schedule is judged on every skill dimension, not only the average. A secondary claim is methodological. We expect that the obvious observational shortcut will give biased attributions, and we can measure how biased. That shortcut is to correlate source exposure with benchmark gains across checkpoints or across runs with different mixtures, which is analogous to reading causes off a cross-sectional survey. The pulse experiments supply the interventional ground truth needed to check it.

The agronomy literature supplies the core concept. Reducing soil nitrogen losses from fertilizer use in global maize and wheat production argues that losses fall mainly by matching release timing to crop demand rather than by cutting rates. Crop studies show that different elements peak at different stages, for example nitrogen, phosphorus and potassium in cotton (Effects of Growth Regulator and Planting Density on Cotton Yield and N, P, and K Accumulation in Direct-Seeded Cotton). The machine-learning translation of these two ideas is a separate uptake curve for each skill, instead of one scalar "data value."

Relation to Prior Work

Three lines of work already make mixtures dynamic.

TiKMiX estimates how raising a domain's weight would lower validation loss, using a gradient-based "Group Influence" score, and re-mixes during training. It reports a 2% average gain across nine benchmarks and 4.8% on ARC-Challenge.

D³ argues that batches influence each other asymmetrically, for example foundational data making later hard data useful. It orders batches using second-order curvature estimates, reports a 4.2% relative perplexity reduction in pretraining and +11.4% on MATH in fine-tuning, and pays an 89% per-step compute overhead.

Does your data spark joy? shows that a crude two-stage schedule works surprisingly well: dropping Common Crawl and upsampling denser domains in the last 10–20% of a 1T-token, 7B run. It reports +6.90 points on MMLU, +8.26 on GSM8K and +6.17 on HumanEval over the base mixture. Gains plateau or regress when the upsampling window grows past 20%.

These papers supply the baselines and the evidence that timing matters. They differ from this proposal in three ways. First, all three steer by loss or by a final benchmark average, never by separate skill trajectories. Second, TiKMiX's and D³'s influence estimates are local approximations from gradients rather than measured interventions. Third, the upsampling paper chooses its switch point by sweeping whole runs.

Two findings constrain the design.

SlimPajama-DC reports that code-heavy mixtures reach lower training loss without better reasoning results ("loss is deceptive"). This is why we score skills with probes rather than with loss. Its static balanced mixture, DC-6, reached 40.0% versus 38.0% average leaderboard accuracy at 1.3B, so a well-built static mixture is a serious baseline.

TiKMiX reports that 1B and 7B models prefer different domains at the same token count. This warns that uptake maps measured on small models may not transfer to larger ones, so we test transfer explicitly.

The competency-modeling literature motivates scoring skills separately and without letting them offset one another. Summing scores across dimensions hides a severe deficit in one of them. Education studies show this concretely: binary internet access fails to predict income while decomposed digital skills do (Can farmers' digital literacy improve income?). The pedagogical-integration literature makes the same point about evaluation that collapses a multi-dimensional curriculum into one exam. SNS-Bench is the machine-learning example of a multi-skill evaluation. It is a single snapshot and says nothing about how the skills develop.

The epidemiology papers are the cautionary template. Examples include trace elements and frailty (Association of dietary intake and whole blood essential trace elements with frailty in older adults) and dietary antioxidants and HPV (An L-shaped link between the composite dietary antioxidant index and human papillomavirus infection). In each, a concurrent snapshot cannot rule out reverse causation or a shared hidden cause. The panel-data estimators in A Practical Guide to Counterfactual Estimators for Causal Inference with Time-Series Cross-Sectional Data are the best observational alternative. We use them as the "cheap estimator" and test them against our interventional ground truth.

Research Program

Stage one builds the uptake map. We train base models on SlimPajama's seven source families: CommonCrawl, C4, GitHub, StackExchange, ArXiv, Wikipedia and Books. FineWeb-Edu is added as an eighth, high-density source (HuggingFaceFW/fineweb). The main models are 410M and 1B parameters trained on about 100B tokens.

At roughly eight checkpoints spaced across training, we fork the model. One fork continues the base mixture for Δ tokens; this is the control. Each other fork replaces half of the mixture with a single source for the same Δ tokens. Δ is roughly 1% of the run. Each pulse is repeated with two data-order seeds.

Skills are grouped into four dimensions: factual knowledge (MMLU), mathematical reasoning (GSM8K), code (HumanEval), and commonsense and science question answering (ARC, HellaSwag, OpenBookQA). Small models sit near zero accuracy on GSM8K and HumanEval. We therefore score every dimension by the model's log-likelihood of the correct answer or reference solution, which moves smoothly long before accuracy does. Accuracy is reported alongside.

The uptake of skill $k$ from source $d$ at checkpoint $t$ is

$$R_k(d,t) = \mathbb{E}\left[s_k\!\left(\theta_{t}^{+d}\right) - s_k\!\left(\theta_{t}^{\text{ctrl}}\right)\right],$$

where $s_k$ is the dimension-$k$ score, $\theta_t^{+d}$ is the model after the $d$-pulse starting at checkpoint $t$, $\theta_t^{\text{ctrl}}$ is the matched control fork, and the expectation is over seeds.

Pulses are randomized and each fork has a clean before and after, so $R$ is an interventional, time-ordered quantity. The design also records whether gains persist. We score forks again after a further Δ tokens of base mixture, which separates durable skill gains from transient memorization that washes out.

Stage two tests the observational shortcut. From the same runs, plus a set of full runs with differently fixed mixtures, we estimate source-to-skill effects in two cheaper ways. The first correlates cumulative source exposure with skill scores across checkpoints. The second applies the fixed-effects counterfactual estimators from the practical-guide paper to the panel of runs and checkpoints. We then report how often each observational estimate gets the sign or ranking of $R_k(d,t)$ wrong. This quantifies how much end-of-run and cross-run comparisons can mislead curriculum claims.

Stage three schedules to the map. We interpolate $R$ smoothly across stages. At each scheduling interval, the mixture puts weight on sources in proportion to their measured uptake for the currently weakest dimension, normalized against that dimension's score under the static mixture. This objective does not let skills offset one another: it improves the minimum dimension rather than the mean. The scheduler is fixed before the evaluation runs. We then test whether a map measured at 410M produces a good schedule at 1B, and at 7B for one final confirmation run.

Baselines

The naive reference is the static SlimPajama mixture held fixed for the whole run. It is trained on the same data, for the same tokens, with the same model and optimizer.

The strongest simple competitor is a static mixture tuned more carefully: the DC-6-style balanced recipe from SlimPajama-DC, which reported 40.0% versus 38.0% for a RedPajama baseline at 1.3B. We give it a small sweep over mixture weights. The number of sweep runs is equal in compute to the pulse budget our method spends, so the comparison charges both sides for their search.

The strongest timing baseline is two-stage domain upsampling as in Does your data spark joy?. Its reported gains are +6.90 MMLU, +8.26 GSM8K and +6.17 HumanEval points at 7B and 1T tokens. We rerun it at our scale, again with a sweep over switch point and window length paid for from the same compute budget.

The strongest dynamic method is TiKMiX-D from TiKMiX, rerun on our sources with its influence computed against a validation set drawn from all four dimensions. D³ is included at 410M only, with its 89% overhead counted in the compute budget, because it orders batches rather than mixing domains.

The key ablation keeps our pulse machinery and scheduler but collapses the four dimensions into their average. This isolates whether scoring each skill separately and optimizing the weakest one is the ingredient that matters.

Beating a baseline means two things at matched total compute, including pulse and sweep costs. The worst-dimension score must be higher, with the margin exceeding the seed-to-seed spread across three training seeds. The average score must be no worse.

Why not just…?

Why not just use two-stage upsampling at the end, since it already captures most of the gain? This is the most serious threat. If the uptake map shows that each source's value is roughly flat until late training and then rises, a two-stage schedule is nearly optimal, and our method reduces to an expensive way of choosing the switch point. The upsampling paper's own finding that gains regress past a 20% window suggests uptake is not a simple step. It does not show which skills lose ground. The measured map settles this directly. If it has no distinct early or middle peaks for any skill, we report that result and recommend the simpler method.

Why not just steer by validation loss, as TiKMiX does? SlimPajama-DC's finding that code-heavy mixtures lower loss without improving reasoning implies that loss-driven schedulers can feed the wrong source. Loss is also one number. It cannot tell a schedule that raises code while starving math from one that raises both evenly. The comparison with TiKMiX-D, reported per dimension, tests whether this matters in practice.

Why not just correlate checkpoint scores with data exposure, or apply panel estimators to existing runs, and skip the costly forks? In machine learning we can intervene. Relying on observational estimates therefore gives up the one advantage we have over the epidemiology studies in the file, whose snapshots could not separate cause from effect. Stage two tests this alternative directly. If the fixed-effects estimators recover the rankings in $R$ with high agreement, they become the recommended cheap method and the forks become a one-time calibration step. That outcome would also be a useful result.

Proposed Experiments and Decisive Checks

The first decisive check is whether stage dependence exists at all. For each skill $k$, we test whether the best source varies across checkpoints beyond what the seed-to-seed spread explains. Stage dependence in the ranking of sources is required. Uptake that merely rises or falls over training while sources keep the same order would not justify a schedule. If no skill's best source changes across checkpoints, the synchronization premise fails, and static or two-stage mixing is sufficient.

The second check is the observational-bias measurement: the fraction of (source, skill, stage) cells where correlation or panel estimates disagree in sign with the pulse estimates.

The third is the head-to-head comparison at 410M and 1B against the baselines above, with worst-dimension and average scores reported.

The fourth is transfer. We run a single 7B schedule built from the 1B map and compare it against 7B two-stage upsampling. This checks TiKMiX's warning that preferences shift with scale.

As a secondary, out-of-distribution multi-skill probe at 7B only, we evaluate instruction-tuned versions on SNS-Bench's four pillars. For reference, the best open model there, DeepSeek-V3, scores 61.90%. This asks whether a more balanced skill profile from pretraining carries over to a practical multi-dimensional task suite. No reported result exists in the file for scheduling methods on SNS-Bench.