Function Preserving Network Expansion

30 concepts22 papers

When training deep neural networks, increasing model capacity—such as widening layers or adding depth—traditionally requires training the new, larger architecture completely from scratch. This process is computationally expensive and discards the valuable representations learned by smaller, pre-trained models.

Function-preserving network expansion solves this inefficiency. The core insight is simple: when expanding a network, we should initialize the larger model such that it is mathematically identical to the smaller model at the moment of transition. By maintaining this functional identity, the larger model can immediately continue training from a warm start without losing previously acquired knowledge or experiencing optimization instability.


The Mechanics of Neuron Splitting

To understand how a network can grow while remaining functionally identical, consider the simplest case: widening a fully connected layer.

Suppose a hidden layer takes an input vector $x$ and passes it through a single neuron to compute an activation $h = \sigma(W_{\text{in}} x)$. This activation is then scaled by an outgoing weight to contribute to the next layer's input: $y = W_{\text{out}} h$.

Standard Layer:
[ x ] ---> ( W_in ) ---> [ h ] ---> ( W_out ) ---> [ y ]

If we want to widen this layer by adding a new neuron, a naive random initialization will alter $y$, disrupting the network's downstream logic. Instead, we can perform a neuron splitting procedure, a concept formalized by Ian Goodfellow et al. in their seminal Net2Net framework.

To split the neuron, we clone its incoming weights exactly, creating two identical hidden units, $h_1$ and $h_2$, which both compute $\sigma(W_{\text{in}} x)$. If we leave the outgoing weights unchanged, the new output becomes $W_{\text{out}} h_1 + W_{\text{out}} h_2 = 2 W_{\text{out}} h$, doubling the original signal. To preserve the original function, we must partition the outgoing weights. If we divide them equally, the new output is:

$$y_{\text{new}} = \frac{1}{2} W_{\text{out}} h_1 + \frac{1}{2} W_{\text{out}} h_2 = W_{\text{out}} h$$

Widened Layer (Function Preserved):
          ---> ( W_in ) ---> [ h_1 ] ---> ( 0.5 * W_out ) ---
[ x ] ---|                                                    |---> [ y ]
          ---> ( W_in ) ---> [ h_2 ] ---> ( 0.5 * W_out ) ---

This simple arithmetic ensures that the network's output remains completely unchanged. As shown by Tao Wei et al. in Modularized Morphing of Neural Networks, this graph-based mapping can be extended beyond simple dense layers to morph any convolutional layer into an arbitrarily complex module while perfectly preserving its mathematical identity.


From Static Matching to Dynamic Equivalence

While dividing weights ensures that a larger model matches the smaller model at step zero, it introduces a new problem: symmetry. If $h_1$ and $h_2$ have identical weights and receive identical gradients, they will update in lockstep during training, functionally acting as a single neuron and wasting the newly added capacity.

To break this symmetry, researchers inject a small amount of noise into the weights. However, adding arbitrary noise breaks the exact function preservation. This tension highlights the difference between static output matching and dynamic trajectory preservation.

To address this, Yuxin Ma et al. introduced the concept of dynamic equivalence in $\mu$pscaling small models: Principled warm starts and hyperparameter transfer. They demonstrated that to prevent training instability, one must scale not just the weights, but also the optimizer states (like those in Adam) and the symmetry-breaking noise according to the network's parametrization.

When done correctly, dynamic equivalence ensures that the widened network follows the exact same functional trajectory during optimization as its narrower predecessor. This allows optimal hyperparameters—such as learning rates—to transfer perfectly from the small model to the large one.


The Geometry of the Loss Landscape

Why does function-preserving expansion work so well from an optimization perspective? The answer lies in the geometry of the loss landscape.

In Embedding Principle of Loss Landscape of Deep Neural Networks, Yaoyu Zhang et al. mathematically proved an embedding principle: the loss landscape of a wider network naturally contains all the critical points (including local and global minima) of its narrower counterparts.

When we perform neuron or channel splitting, we are not just guessing a good initialization; we are precisely mapping a critical point from a low-dimensional loss landscape into a high-dimensional one. As further elaborated by Jiahan Zhang et al. in Embedding Principle of Homogeneous Neural Network for Classification Problem, this mapping preserves the optimization paths (KKT points), meaning the expanded network inherits the low-complexity solutions and stable convergence properties discovered by the smaller model.


Applications in Modern Architectures

This paradigm has become a cornerstone for accelerating the training of massive models and adapting architectures efficiently across various domains:

  • Large Language Models: Training Transformers from scratch is incredibly resource-intensive. Frameworks like CompoundGrow (Xiaotao Gu et al.), Staged Training (Sheng Shen et al.), and bert2BERT (Cheng Chen et al.) use function-preserving operators to scale BERT and other LLMs across width, depth, and sequence length simultaneously, cutting pre-training compute costs by up to 50%.
  • Data-Scarce Regimes: When training data is limited, large models often overfit or fail to converge. Peihao Wang et al. demonstrated in Data Efficient Neural Scaling Law via Model Reusing that expanding a model width-wise while preserving functionality minimizes knowledge loss, allowing massive models to scale effectively even on tiny datasets.
  • Neural Architecture Search (NAS): Traditional NAS requires training thousands of candidate networks from scratch. By using function-preserving transformations, frameworks like EAS (Han Cai et al.) and FNA (Jiemin Fang et al.) allow reinforcement learning agents or search algorithms to "grow" networks dynamically, reducing search times by orders of magnitude.

By shifting the perspective from "training isolated models" to "continuously growing a single functional asset," function-preserving expansion bridges the gap between small-scale experimentation and large-scale deployment.

Key Foundational Work

Net2Net: Accelerating Learning via Knowledge Transfer (2016) — contributed 2 concept elements to this direction.
Net2Net: Accelerating Learning via Knowledge Transfer Tianqi Chen (2016) — contributed 1 concept elements to this direction.
Modularized Morphing of Neural Networks (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_function_preserving_network_expansion_8167cadf,
  title = {function-preserving network expansion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/function-preserving-network-expansion/rdr_8167cadf4e2a415d8c816b449674bfb9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_function_preserving_network_expansion_8167cadf_v1,
  title = {function-preserving network expansion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/function-preserving-network-expansion/rdr_8167cadf4e2a415d8c816b449674bfb9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake