Efficient Model Expansion through Progressive Growth

This direction is merged into another direction: stochastic symmetry breaking (symmetry-breaking initialization). Its last version stays readable and citable here.

Version 1 of 112 papers18 conceptsfrozen 2026-10-04

Training massive language models from scratch is increasingly viewed as a "wasteful" use of computational resources. When a researcher wants to move from a 1-billion parameter model to a 10-billion parameter model, the standard practice has been to discard the smaller model and start over.

A growing body of research suggests a more efficient alternative: progressive model expansion. By "recycling" the knowledge of smaller pretrained models to initialize larger ones, researchers can slash pretraining costs by 35% to 57% while maintaining state-of-the-art performance.

The Challenge of Function Preservation

The primary goal of model expansion is to increase a network's width (neurons) or depth (layers) without losing the knowledge already stored in its weights. If you simply add random neurons to a pretrained network, the model’s output changes instantly, causing a "performance dip" that requires significant compute to repair.

To avoid this, researchers use Function Preserving Transforms (FPTs). As explored by Michael Painter in Towards a More Complete Theory of Function Preserving Transforms, these transforms ensure that the expanded model’s initial output is mathematically identical to the original model’s output. For example, when widening a layer, one might duplicate existing neurons but scale their output weights so the total sum remains the same.

The Symmetry Trap

While function preservation keeps the model's performance stable, it introduces a secondary problem: parameter symmetry.

If you expand a model by creating exact clones of existing neurons, those clones will receive the exact same gradients during training. They will move in lockstep, learning the same features and effectively wasting the new capacity. Cheng Chen and colleagues, in their work on bert2BERT, note that this symmetry creates "redundant learning cycles" that slow down convergence.

To make expansion useful, the model must break this symmetry so that new neurons can specialize into different features.

Mechanisms for Breaking Symmetry

The core innovation in recent expansion frameworks is the balance between preserving the signal and forcing the parameters to diverge.

  1. Asymmetric Initialization: Instead of identical cloning, Yite Wang and the authors of LEMON propose initializing replicated neurons with unequal output weights that still sum to unity. This ensures the initial function is preserved, but the differing weights ensure that gradients will immediately begin to pull the neurons in different directions.
  2. Masked Fading: Yiqun Yao introduces Masked Structural Growth (MSG) in 2x Faster Language Model Pre-training via Masked Structural Growth. This method decouples initialization from function preservation by using an external mask to gradually "fade in" new, randomly initialized neurons. Because the neurons start random, they are never in symmetry, but the mask prevents their randomness from disrupting the model's existing knowledge.
  3. Optimizer Interventions: Even with clever initialization, the "memory" of the optimizer can keep neurons tethered. Qifan Yu and the SPARKLING team suggest resetting optimizer states or applying localized learning rate re-warmups for new parameters. This "symmetry breaking in the optimizer" forces new weights to diverge from their parents.

Why It Works: Expressivity and Chaos

Beyond just saving compute, these initialization strategies may actually improve the underlying health of the network. Standard deep networks often suffer from "dying ReLUs" or highly correlated signals that reduce expressivity.

Research into Random Asymmetric Anti-correlated Initialization (RAAI) suggests that by using anti-correlated weights and positive-biased distributions, we can push networks into a "chaotic phase." This phase prevents neurons from becoming inactive and ensures that the expanded capacity is fully utilized to learn complex functions. By carefully managing how we grow a model, we aren't just making it bigger—we are ensuring it remains an expressive, efficient learner.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_efficient_model_expansion_through_progre_b86321e7,
  title = {Efficient Model Expansion through Progressive Growth},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-model-expansion-through-progressive-growth/rdr_b86321e736c945b984c71b1205bfd134},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_efficient_model_expansion_through_progre_b86321e7_v1,
  title = {Efficient Model Expansion through Progressive Growth},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-model-expansion-through-progressive-growth/rdr_b86321e736c945b984c71b1205bfd134/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}