Training massive language models from scratch is increasingly viewed as a "wasteful" use of computational resources. When a researcher wants to move from a 1-billion parameter model to a 10-billion parameter model, the standard practice has been to discard the smaller model and start over.
A growing body of research suggests a more efficient alternative: progressive model expansion. By "recycling" the knowledge of smaller pretrained models to initialize larger ones, researchers can slash pretraining costs by 35% to 57% while maintaining state-of-the-art performance.
The Challenge of Function Preservation
The primary goal of model expansion is to increase a network's width (neurons) or depth (layers) without losing the knowledge already stored in its weights. If you simply add random neurons to a pretrained network, the model’s output changes instantly, causing a "performance dip" that requires significant compute to repair.
To avoid this, researchers use Function Preserving Transforms (FPTs). As explored by Michael Painter in Towards a More Complete Theory of Function Preserving Transforms, these transforms ensure that the expanded model’s initial output is mathematically identical to the original model’s output. For example, when widening a layer, one might duplicate existing neurons but scale their output weights so the total sum remains the same.
The Symmetry Trap
While function preservation keeps the model's performance stable, it introduces a secondary problem: parameter symmetry.
If you expand a model by creating exact clones of existing neurons, those clones will receive the exact same gradients during training. They will move in lockstep, learning the same features and effectively wasting the new capacity. Cheng Chen and colleagues, in their work on bert2BERT, note that this symmetry creates "redundant learning cycles" that slow down convergence.
To make expansion useful, the model must break this symmetry so that new neurons can specialize into different features.
Mechanisms for Breaking Symmetry
The core innovation in recent expansion frameworks is the balance between preserving the signal and forcing the parameters to diverge.
- Asymmetric Initialization: Instead of identical cloning, Yite Wang and the authors of LEMON propose initializing replicated neurons with unequal output weights that still sum to unity. This ensures the initial function is preserved, but the differing weights ensure that gradients will immediately begin to pull the neurons in different directions.
- Masked Fading: Yiqun Yao introduces Masked Structural Growth (MSG) in 2x Faster Language Model Pre-training via Masked Structural Growth. This method decouples initialization from function preservation by using an external mask to gradually "fade in" new, randomly initialized neurons. Because the neurons start random, they are never in symmetry, but the mask prevents their randomness from disrupting the model's existing knowledge.
- Optimizer Interventions: Even with clever initialization, the "memory" of the optimizer can keep neurons tethered. Qifan Yu and the SPARKLING team suggest resetting optimizer states or applying localized learning rate re-warmups for new parameters. This "symmetry breaking in the optimizer" forces new weights to diverge from their parents.
Why It Works: Expressivity and Chaos
Beyond just saving compute, these initialization strategies may actually improve the underlying health of the network. Standard deep networks often suffer from "dying ReLUs" or highly correlated signals that reduce expressivity.
Research into Random Asymmetric Anti-correlated Initialization (RAAI) suggests that by using anti-correlated weights and positive-biased distributions, we can push networks into a "chaotic phase." This phase prevents neurons from becoming inactive and ensures that the expanded capacity is fully utilized to learn complex functions. By carefully managing how we grow a model, we aren't just making it bigger—we are ensuring it remains an expressive, efficient learner.