Initializing Small Models via Pretrained Weight Inheritance

29 concepts22 papers

When we need a lightweight neural network for edge deployment, our default instinct is often to design a compact architecture and train it from scratch. However, an enduring paradox in deep learning is that small models trained this way rarely match the performance of larger models, even when given identical amounts of data.

To bridge this gap, a powerful paradigm has emerged: pruning-based initialization. Instead of starting from random noise, we initialize a smaller model by "carving out" a subset of weights from a massive, pre-trained network. By treating pruning not merely as a post-processing compression step but as a foundational initialization strategy, we allow compact networks to inherit an optimized feature space that they could never discover on their own.

The Optimization Paradox of Small Models

Why does training a small model from scratch underperform? The answer lies in the geometry of the loss landscape. Large models benefit from over-parameterization; their high-dimensional landscapes contain numerous smooth paths to high-quality local minima. Small models, by contrast, operate in constrained spaces riddled with poor local minima and optimization traps.

A wide body of research confirms that leveraging a large model's optimization journey yields superior results. Jian-Hao Luo et al. demonstrated this in their foundational work on ThiNet, showing that a pruned network significantly outperforms an identical architecture trained from scratch. This phenomenon spans domains: Yasutoshi Ida et al. observed it in residual networks via Network Implosion, Yulun Zhang et al. verified it for image super-resolution, and Miao Zhang documented in a comprehensive survey that sparse topologies consistently fail to replicate the performance of inherited weights when initialized randomly.

Traditional Pipeline:
[Random Init] ───────> Train Small Model ───────> Suboptimal Performance

Weight Inheritance Pipeline:
[Large Pretrained Model] ───(Prune/Carve)───> [Inherited Init] ───(Fine-tune)───> Superior Performance

One might wonder if traditional Knowledge Distillation (KD)—where a small student model mimics a large teacher—solves this problem. However, KD still forces the student to navigate a treacherous loss landscape from a random starting point. In contrast, Dhanush Bekal et al. showed that physically removing layers from a pre-trained model and fine-tuning the remainder outperforms imitation-based distillation. Direct structural reduction preserves the complex, high-quality features captured during large-scale pre-training more effectively than a student can copy them.

What Exactly is Inherited?

When a small model inherits weights from a pre-trained giant, it is not just receiving good numerical starting values; it is inheriting structural knowledge and emergent capabilities.

In Initializing Models with Larger Ones, Zhiqiu Xu et al. explored this mechanism by sub-sampling weights from large-scale vision and language models. They discovered that the inherited weights carry complex structural properties—such as specific, highly coordinated attention patterns—that typically only emerge in high-capacity models after extensive training. A small model initialized randomly may never develop these patterns due to capacity constraints, but it can successfully maintain and utilize them if they are provided at birth.

Similarly, Haokun Lin et al. applied this concept to vision-language models with MoPE-CLIP, demonstrating that inheriting established structural representations allows compact models to outperform larger architectures trained from scratch on vastly more data. Theoretical backing from Mao Ye 0006 et al. confirms that these inherited parameters contain critical information that cannot be replicated by re-training a sparse topology, proving that the specific weight values matter just as much as the pruned architecture itself.

Mechanics of Carving Out a Subnetwork

To successfully initialize a smaller model, we must decide which dimensions of the large model to preserve. Researchers have developed several strategies to extract these high-value subnetworks across different structural granularities:

1. Layer-Level Slicing

The simplest approach is to drop entire blocks or layers. Weidong Chen et al. introduced Vesper for speech emotion recognition, initializing a compact model by selecting a subset of layers from a large pre-trained backbone at uniform intervals.

To be more surgical, data-driven metrics can identify redundancy. Wei Liu et al. developed CoMFLP, which computes activation correlation matrices across layers to find and eliminate those that contribute redundant information, providing a highly stable initialization for subsequent fine-tuning.

2. Fine-Grained Structural Slicing

For modern Transformer architectures, pruning must often happen across multiple dimensions simultaneously. Chaofan Tao et al. designed the SIMPLE framework to compress generative language models by targeting attention heads, feed-forward network dimensions, and the core hidden dimension itself.

To automate this process, Yehui Tang utilized learnable masks during the pre-training phase to dynamically determine the optimal subnetwork boundaries. This mask-driven inheritance ensures that the resulting tiny language model starts with a significantly lower initial training loss and achieves superior final convergence.

3. Auxiliary Side-Networks

Weight inheritance can also be used to initialize entirely separate, lightweight companion networks. Jaemin Cho proposed Ladder Side-Tuning (LST), where an auxiliary side network is initialized by pruning a large pre-trained backbone based on importance metrics like Fisher Information or weight magnitude. Because the side network inherits pre-trained knowledge from the backbone at the outset, it avoids the slow burn of random initialization while drastically reducing backpropagation memory costs.

The Efficiency Dividend

The ultimate justification for pruning-based initialization is its extreme computational efficiency. While training a small model from scratch requires a full, expensive training run, fine-tuning an inherited subnetwork requires only a fraction of the time and data.

Compute Budget Comparison for Small Model Training:

From Scratch:  ████████████████████████████████ 100%
Inherited:     █ 3% (e.g., Sheared LLaMA)

A striking example of this is Sheared LLaMA by Mengzhou Xia et al.. By using targeted structured pruning to initialize smaller language models from LLaMA2, the authors produced state-of-the-art 1.3B and 2.7B models using as little as 3% of the compute budget typically required to train models of that size from scratch. The models inherited existing linguistic and factual knowledge, bypassing the costly early phases of pre-training.

Furthermore, this initialization advantage minimizes downstream adaptation friction. Jingxuan Zhou et al. demonstrated in CUT that freezing weights during sensitivity evaluation preserves high-quality features, allowing for extreme sparsity levels with minimal recovery time. By starting the optimization process at a highly advanced point, pruning-based initialization fundamentally changes the economics of deploying efficient, high-performance deep learning models.

Key Foundational Work

ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression (2017) — contributed 1 concept elements to this direction.
Architectures for deep neural network based acoustic models defined over windowed speech waveforms (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_pruning_based_model_initialization_e56010ed,
  title = {pruning-based model initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pruning-based-model-initialization/rdr_e56010eda90647718594b9f44a26559b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_pruning_based_model_initialization_e56010ed_v1,
  title = {pruning-based model initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pruning-based-model-initialization/rdr_e56010eda90647718594b9f44a26559b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}

See something wrong? Report a mistake