Language models are pretrained on a mix of sources: web crawl, books, code, papers, forums. Practitioners usually sample each source in proportion to its size or a hand-tuned multiplier. Finance knows this "weight by size" policy fails in a particular way. When holdings are weighted by market value, the largest and most correlated assets end up driving almost all of the portfolio's movement. Risk parity fixes this by choosing weights so that each asset contributes an equal, or a deliberately budgeted, share of total variance. This note argues the same failure happens inside a training step. A few large, mutually aligned domains end up controlling the direction of the update, and redundant signal that is already saturated keeps being paid for with tokens.
The proposal treats per-domain gradients as the "assets" of a portfolio. Every few hundred steps, on the model actually being trained, we measure how much each domain contributes to the squared size of the combined update, counting the overlap between domains. We then reset sampling weights so these contributions match target budgets. The budgets come from fitted diminishing-returns curves: domains with more predicted improvement remaining get larger budgets. Existing reweighting methods chase a noisy signal of expected gain, such as excess loss, validation-gradient alignment, or rate of improvement. This proposal instead does most of its allocation through covariance, which is estimated more stably. We think that difference explains why dynamic mixtures have sometimes hurt in pretraining, and that it can be fixed.
Core Claim
Define $g_i(t)$ as the mean gradient of domain $i$ at step $t$, and $w$ as the sampling weights on the simplex. The combined update $\sum_i w_i g_i$ has squared norm $w^\top G w$, where $G_{ij} = \langle g_i, g_j \rangle$ is the domain gradient Gram matrix, smoothed over recent steps. Domain $i$'s risk contribution is $RC_i = w_i (G w)_i$, and these contributions sum to the total. We choose weights by solving the risk-budgeting condition
$$ w_i (G w)_i = b_i \, w^\top G w \quad \text{for all domains } i, $$
where $b_i \ge 0$ sums to one. The budget is set from a per-domain saturating fit $L_i(n) = E_i + A_i n^{-\alpha_i}$, with $n$ the tokens seen from domain $i$, $E_i$ its irreducible loss, and $A_i, \alpha_i$ fitted constants. We take $b_i \propto -\partial L_i / \partial n$ at the current $n$, the predicted marginal return of one more token.
The power-law form is the same Chinchilla-style fit that Velocitune uses to extrapolate target losses. Because $G$ is positive semidefinite, the system has a unique positive solution, and standard risk-parity solvers compute it cheaply (riskParityPortfolio).
The claim has two parts, and each can fail on its own. First, the off-diagonal structure of $G$ carries information that per-domain signals miss. Domains whose gradients point along the dominant shared direction are partly redundant, and the covariance term charges them for it. Second, because the allocation leans on second moments rather than a noisy expected-gain estimate, it can run online on the large model without the degradation DOGE reports when it applies its dynamic proxy schedule directly.
Where This Sits
Several papers motivate the work. Portfolio of Risk Premia shows that size-based allocations bundle correlated exposures, and that weighting decomposed drivers instead cut volatility from 8.4% to 2.8% at similar return. That is the structural analogy we import. Capturing the Temporal Dependence of Training Data Influence finds a long mid-training "influence saturation" phase in which examples resembling earlier ones have near-zero marginal effect. We read this as redundancy, which is exactly what a covariance-aware allocation is designed to detect. On the importance of pre-training data volume for compact language models, where 40x more data moved FQuAD F1 only from 68.50 to 69.76, and FineWeb establish that the marginal value of tokens is strongly concave and differs by source. That is the justification for letting fitted saturation curves set the budgets.
The closest methods fall into two groups. The first group learns static weights on a small proxy model and transfers them to the large one. DoReMi runs group distributionally robust optimization, a minimax over domains, on excess loss relative to a reference model. DOGE upweights domains whose gradients align with a validation gradient. Both average away the trajectory, and both use a first-moment "which domain helps most" signal.
The second group reweights online. Velocitune prioritizes domains with remaining progress relative to a scaling-law target. GRAPE prioritizes targets with the slowest rate of improvement. Neither accounts for overlap between domains. If two domains teach the same thing, both receive full credit for their lagging progress.
Our method borrows Velocitune's saturation fit, but only to set budgets. Allocation then runs through $G$. We need no reference model (unlike DoReMi) and no validation set (unlike DOGE), although budgets could be tied to target tasks.
Two papers constrain the design. DOGE's negative result on dynamic schedules is a direct threat, and one of our decisive checks targets it. Velocitune's finding that raw loss-distance weighting locks onto intrinsically noisy domains warns against budgets based on current loss rather than marginal return.
We leave aside token-level selection such as ESLM, which uses a Value-at-Risk cutoff on the token set. It is complementary, and folding it in would blur the test. Scaling-law mixers such as CAMEL and ShapleyLaw optimize a predicted expected loss with a static recipe and never model variance contributions. We return to their proxy-transfer question only as a secondary analysis.
Research Program
The work has three parts, and the first can kill the idea cheaply.
Diagnostic study. On standard pretraining runs, we log per-domain gradients by routing each microbatch to a single domain, which is already natural under gradient accumulation. We project them to 1024 dimensions with a fixed random projection, following the memory-saving recipe in the influence-embedding work above, and track $G$ over training. We then compute a concentration ratio: each domain's share of $w^\top G w$ divided by its token share.
The hypothesis follows the finance analogy. Under size-proportional sampling, the large web domains take a share of the update well above their token share, and this concentration grows in the mid-training saturation phase as domain gradients become more collinear. If off-diagonal cosines turn out to be small throughout, then $G$ is effectively diagonal, and the method reduces to inverse-magnitude weighting. We would report that and stop claiming a covariance effect.
The method. We re-estimate $G$ with an exponential moving average and refit the saturation curves every 100 steps, the update period GRAPE uses at about 15% overhead. We solve the budgeting problem and blend the new weights into the old ones.
One safeguard is needed. A saturated domain with a tiny gradient could otherwise be heavily upweighted just to meet its budget, which is the finance problem of levering up a low-volatility asset. The saturation-derived budget is what prevents this: once marginal return collapses, $b_i \to 0$. We also cap each weight at a multiple of its natural share.
Analysis. We ask whether risk-budgeted weights computed on a small model rank domains the same way as those computed on the large model. CAMEL's evidence that domain utility depends on model size suggests they will not. If the ranks do hold, the method could also be run proxy-style.
Baselines
Size-proportional sampling is the natural mixture of SlimPajama or The Pile. It is the "capital-weighted" policy the proposal criticizes. Uniform sampling across domains is included as well, because it is what most reweighting papers compare against. These two establish the do-nothing floor.
DOGE is the strongest existing competitor on SlimPajama. It is target-aware, cheap, and reported on the same corpus and model sizes we can afford. DOGE reports that its 684M model improves 5-shot accuracy on the six reasoning tasks (COPA, SciQ, LogiQA, PIQA, WiC, WinoGrande) by 1.7 percentage points on average over uniform and DoReMi. It also reports lower perplexity on 5 of 7 domains while using 10x fewer FLOPs than DoReMi. We will run DOGE with its released-paper setup: a 60M–124M proxy, then time-averaged weights for the base model. The base model will use the same tokenizer, token budget, architecture, and learning-rate schedule as ours.
DoReMi is included on The Pile. DoReMi reports +6.5 points average few-shot accuracy and 2.6x faster time-to-baseline at 8B. We will not reproduce that scale; we compare at our largest affordable size with a 280M-style proxy. Velocitune and GRAPE are the online competitors. We run them on the base model with the same update period, so reweighting frequency and overhead are matched.
Tuning is matched across methods. Every method gets the same budget of hyperparameter trials over its own knobs: step sizes for DRO-style updates, smoothing for ours, and proxy size where relevant. Every method's compute overhead is charged against its token budget. Beating DOGE means a larger average gain on its six-task suite at 684M, plus a lower worst-domain perplexity, at equal total FLOPs including overhead, holding across three seeds.
Three ablations isolate the proposal's ingredients. The diagonal ablation replaces $G$ with its diagonal. This is inverse-volatility risk parity, and it tests whether covariance matters. The equal-budget ablation sets $b_i = 1/k$, which tests whether the diminishing-returns budgets matter. The static-average ablation averages our own dynamic weights and trains with them fixed, which tests whether online allocation is actually the source of any gain.
Why not just…?
Why not just use DOGE or DoReMi weights and skip the dynamics? Static transferred weights are the current strongest practice, and DOGE's own authors found that dynamic schedules hurt their base model. That is a real threat. The static-average ablation settles it directly. If averaged risk-budgeted weights match the online version, the contribution shrinks to a different way of computing a static mixture, and we will say so. The prediction that separates the approaches concerns where the gain shows up. DOGE's failure should come from applying the fluctuating first-moment signal of a distorted proxy to a different model. Our weights are computed on the model itself from a smoothed second-moment matrix, so they should avoid that failure. The gain should be largest in the mid-training saturation phase that static averages cannot track.
Why not just normalize each domain's gradient so every domain contributes equally? Per-domain gradient normalization, or inverse-magnitude weighting, is the simplest variance-aware fix, and it may capture most of the benefit. It is exactly our diagonal ablation. If off-diagonal structure is weak, which the diagnostic study measures first, this alternative wins, and we would recommend it over the full method. The case for covariance rests on a specific prediction: two domains that share a direction, such as C4 and CommonCrawl, should jointly receive less weight under full $G$ than under the diagonal. The benefit should appear on the domains that the shared direction crowds out.
Why not just smooth Velocitune's or GRAPE's weights with a moving average? That could stabilize dynamic reweighting without any covariance machinery. The smoothing constant becomes part of both methods' tuning budget, so this is tested fairly. What smoothing cannot do is remove double counting. If two redundant domains both lag, velocity- and improvement-based rules reward both. Our method charges them for their overlap. A smoothed Velocitune matching us would indicate that stability, not redundancy accounting, drove the result.
Proposed Experiments
The main testbed is SlimPajama's 7 domains at 124M, 350M, and 684M parameters. This is the corpus of SlimPajama-DC, and the 684M size matches DOGE. We evaluate per-domain and worst-domain held-out perplexity, 5-shot accuracy on DOGE's six reasoning tasks, and tokens to reach the size-proportional baseline's final average accuracy. A second testbed on The Pile's 22 domains at the largest affordable size checks whether conclusions hold with many more, overlapping domains, which is where covariance should matter most.
The decisive checks follow the claim's two parts. First, the diagnostic must show meaningful off-diagonal structure in $G$ and rising update concentration during the saturation phase. Otherwise the covariance story fails. Second, full risk budgeting must beat the diagonal ablation and the smoothed online baselines at matched FLOPs. Otherwise overlap accounting adds nothing. Third, online risk budgeting must not show the degradation relative to its own static average that DOGE saw. Otherwise the stability argument fails. Fourth, equal budgets must underperform saturation-derived budgets, particularly late in training. Otherwise the diminishing-returns component is decorative.
A result where the full method beats DOGE on average accuracy but not on worst-domain perplexity would count as partial success. We would report it as such, since risk parity's promise is balance across domains, not just a higher mean.
References
- Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training
- riskParityPortfolio: Design of Risk Parity Portfolios
- DOGE: Domain Reweighting with Generalization Estimation
- Portfolio of Risk Premia: A New Approach to Diversification
- Capturing the Temporal Dependence of Training Data Influence
- On the importance of pre-training data volume for compact language models
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
- GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining
- ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
- Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization
- ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws
- SlimPajama-DC: Understanding Data Combinations for LLM Training