Progressive model expansion trains a small network first and then enlarges it mid-run, adding width or depth, so that cheap early steps do the coarse learning. When it works it saves 20 to 50 percent of pretraining compute. The fragile moment is the jump itself. Methods disagree about what a good jump looks like. Some insist the enlarged model must compute exactly the same function as the small one. Others argue that exact copies trap the new neurons into learning nothing new, and that it is better to accept a brief loss spike. Neither camp has a measurement that predicts which jump will train fastest afterward.
A separate literature asks why pretrained weights speed up later training, and gives a candidate answer. Pretrained models start with high gradient coherence: gradients from different examples point the same way, so they add up instead of cancelling. This note proposes that a grown model is just a model "pretrained" by its smaller ancestor. The speedup a growth step buys should therefore be governed by how much coherence the new parameters inherit, not by how small the loss is right after the jump. We will test that claim, and if it holds, use a cheap coherence probe to choose how to initialize the new parameters at each growth step.
Core Claim
Our hypothesis is that, across expansion methods, the gradient coherence of newly added parameters in the first few hundred steps after growth predicts how many tokens the grown model needs to catch up with a model trained from scratch at full size. We expect it to predict this better than the loss right after expansion or the activation-scale statistics current methods target.
We measure coherence with the gradient signal-to-noise ratio (GSNR). For a parameter group $G$,
$$ \mathrm{GSNR}(G) = \frac{1}{|G|}\sum_{j\in G}\frac{\left(\mathbb{E}_x[g_j(x)]\right)^2}{\mathrm{Var}_x[g_j(x)]}, $$
where $g_j(x)$ is the gradient of the loss on example $x$ with respect to parameter $j$, and the expectation and variance run over examples in a batch. High GSNR means per-example gradients agree.
The hypothesis explains an open contradiction. SPARKLING widens models mid-pretraining by keeping hidden-state scale consistent and resetting the optimizer state of new parameters. It reports that tolerating an initial loss jump gives lower final loss than strict function preservation. In contrast, LEMON enforces exact function identity, using unequal output weights to break neuron symmetry. Masked Structural Growth (MSG) also enforces it, by masking new units to zero and fading them in. Our account says the function-preserving camp wins on immediate loss and can lose on inherited coherence. Copied neurons receiving identical gradients ("symmetry lock") have redundant gradients that carry no new signal. Zero-masked units may receive gradients that cancel across examples, recreating the "tug-of-war" phase of random initialization.
Prior Work and How It Positions This
Two papers motivate the mechanism. Language Pretraining Gives Structured Forecasters a Sequential Prior shows that pretrained models have much higher early gradient coherence and an effective 6.8x data multiplier on time-series forecasting. Frozen Language Models Are Gradient Coherence Rectifiers in Vision Transformers raises Vision Transformer GSNR by over 10% using an auxiliary frozen language-model block. Both treat coherence as an explanation of transfer. Neither touches architectural growth.
Rethinking ImageNet Pre-training shows that scratch training catches up given 3x to 6x longer schedules. It constrains our framing: what growth can sell is speed, not final quality, so our primary metric is tokens-to-parity.
The growth methods supply the operators we compare. MSG, LEMON, and SPARKLING are described above. Warmstarting for Scaling Language Models zero-pads, shrinks by 0.4, and adds noise. Variance Transfer rescales new weights and gives them separate learning rates. All of them judge a growth step by loss curves, activation scale, or weight norms. None measures coherence across the transition.
For timing, FRAGrow schedules convolutional-network growth from the train–validation gap. That signal is nearly uninformative in single-epoch language-model pretraining, where the model never overfits. RAPTR notes that its theory explains why loss does not spike at stage changes but not why loss then drops quickly. Inherited coherence is a candidate answer we can check on RAPTR transitions too.
Research Program
The work has three stages, and each gates the next.
Stage 1: measurement. We train decoder-only Transformers on FineWeb-Edu, the corpus used by the model-family expansion work. A seed model of roughly 125M parameters is grown to roughly 350M, and in a second tier to roughly 1B. The operators are MSG masking, LEMON asymmetric copying, SPARKLING scale-consistent copying with asymmetric reset, shrink-and-perturb, and naive zero-padding. We sweep growth time and the symmetry-breaking magnitude within each operator, giving on the order of forty growth configurations per tier. For every run we log several quantities:
- per-layer GSNR, separately for inherited and new parameters, every 50 steps for 2,000 steps after growth;
- loss immediately after growth;
- hidden-state RMS;
- tokens-to-parity with a scratch run at full size, defined as the tokens at which the grown run's validation loss first matches the scratch run's loss at the end of training.
Stage 2: prediction. We fit rank correlations between each early signal and tokens-to-parity across configurations, holding out entire operators to test generalization. The hypothesis survives only if new-parameter GSNR over the first 500 steps predicts tokens-to-parity better than immediate loss and RMS deviation, including on held-out operators.
Stage 3: a method, built only if Stage 2 succeeds. At each growth step we generate a small set of candidate initializations for the new parameters, varying operator family, perturbation scale, and copy ratio. We run each candidate for K probe steps on a shared set of batches, with K around 100, and keep the one with the highest new-parameter GSNR. The probe costs well under one percent of training compute and is charged to the method.
We also test a layer-wise variant. Gradient coherence is known to concentrate near pretrained anchor layers, and ADEPT found that deep layers are the most adaptable. So the probe may also choose which layers receive extra width. This variant is secondary and is reported separately.
Baselines
The naive reference is a full-size model trained from scratch on the same FineWeb-Edu token budget, with its learning rate tuned on the same grid as every other method. Every speedup is measured against this run. A second weak reference is naive zero-padding growth, which shows how much any growth method recovers at all.
The strongest existing competitor is SPARKLING. It is the most recent mid-stage width-growth method and is evaluated at the largest scale: it reports a 20% to 35% FLOP reduction, up to 1.49x wall-clock speedup, and an average zero-shot score of 62.55 versus 60.89 for the scratch baseline on Dense Transformers and OLMoE trained up to 200B tokens. Our budget is smaller, so we reimplement it at our scale with its recommended re-warmup and reset settings, plus an equal-size tuning grid. To beat it, the coherence-selected initialization must reach scratch parity in fewer total tokens, probe cost included, across three seeds, without lower zero-shot scores.
MSG is the strongest function-preserving competitor. It reports 1.4x speedup on GPT-2, and we run it with its published masking ramp. LEMON is run with its fast learning-rate decay. Shrink-and-perturb is run at λ = 0.4. All competitors get the same seed model, the same growth time grid, and the same number of tuning runs.
The key ablation replaces coherence as the selection criterion while keeping everything else fixed: same candidate set, same probe steps. One version picks the lowest probe loss, the other picks a random candidate. If probe-loss selection matches coherence selection, the mechanism adds nothing beyond a short look-ahead.
Why not just…?
Why not just trigger growth and choose initialization by watching the loss? This is the most serious alternative. Selecting the candidate with the lowest loss after K steps is exactly our key ablation. If it ties, we report that coherence is an explanation but not a better control signal, and drop the Stage 3 method claim. Our reason to expect a gap is SPARKLING's finding that low immediate loss under function preservation can coincide with worse final loss. Stage 2 tests directly whether short-horizon loss inherits that blind spot.
Why not just tune SPARKLING's hyperparameters harder? A grid over perturbation scale and re-warmup might find the same initializations our probe finds. We therefore give the tuned-SPARKLING baseline a tuning budget equal to our candidate count times full runs. Our method needs only K probe steps per candidate. The comparison is then about whether a cheap in-run probe recovers what expensive offline search finds. If both match, the contribution is cost, not quality, and we will say so.
Why not just train from scratch longer, as Rethinking ImageNet Pre-training suggests? That paper shows that parity eventually arrives but needs 3x to 6x more iterations. Our metric is tokens-to-parity, so the scratch run is the yardstick rather than a competitor that can win by running longer. The claim at risk is whether growth helps at all at our scale. The zero-padding reference and the published growth baselines settle that.
Decisive Checks
The first decisive check is Stage 2's held-out-operator correlation. On the 350M tier, new-parameter GSNR must rank configurations by tokens-to-parity with Spearman correlation clearly above that of immediate post-growth loss. A pre-registered margin of at least 0.2 must hold on operators excluded from fitting. Failure ends the method stage and leaves a negative measurement result, which is still worth publishing given the contradiction between the function-preserving and symmetry-breaking camps.
The second check is that coherence selection beats probe-loss selection and tuned SPARKLING in tokens-to-parity at both tiers. Downstream evaluation uses the zero-shot suite of ARC-e, ARC-c, HellaSwag, PIQA, WinoGrande, LAMBADA, OBQA, and SciQ used by SPARKLING and the model-family work, plus WikiText-103 perplexity.
The third check, as a mechanism test, is RAPTR stage transitions. Its rapid post-transition loss drops should coincide with rises in the GSNR of newly activated layers. If they do not, the account is incomplete.
References
- SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
- LEMON: Lossless model expansion
- Masked Structural Growth for 2x Faster Language Model Pre-training
- Language Pretraining Gives Structured Forecasters a Sequential Prior
- Frozen Language Models Are Gradient Coherence Rectifiers in Vision Transformers
- Rethinking ImageNet Pre-Training
- Warmstarting for Scaling Language Models
- Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation
- When to Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks
- Efficient stagewise pretraining via progressive subnetworks
- Efficient Construction of Model Family through Progressive Training Using Model Expansion
- ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning