Initializing Student Networks with Pretrained Teacher Weights

133 concepts94 papers1 proposal

Knowledge distillation is a classic paradigm for model compression: a large, high-performing "teacher" model guides a smaller, more efficient "student" model by training it to match the teacher’s outputs or intermediate representations. Traditionally, the student model starts as a blank slate, initialized with random weights.

However, starting from scratch creates a massive optimization hurdle. The student must simultaneously learn fundamental feature extractions and attempt to align its internal representation space with a highly optimized teacher.

To bypass this chaotic early phase of training, a powerful strategy has emerged: teacher-student weight inheritance. Instead of random initialization, the student model inherits weights directly from the pre-trained teacher before distillation begins. This "warm start" fundamentally alters the dynamics of knowledge transfer.


To understand why weight inheritance works, consider the optimization landscape of a neural network. A randomly initialized student starts in a distant, unoptimized valley of the loss landscape. To mimic the teacher, it must navigate complex non-convex terrain, climbing over high ridges to find a parameter region that yields similar representations.

Random Initialization:
[Student Start] ----(High Loss Terrain)----> [Teacher's Representation Space]

Weight Inheritance:
[Teacher Model] --(Weight Mapping)--> [Student Start] ~~~(Smooth Alignment)~~~> [Optimized Student]

By directly seeding the student with the teacher's parameters, we drop the student directly into the neighborhood of a proven solution. Yevgen Chebotar et al. demonstrated this effect in speech recognition systems within Distilling Knowledge from Ensembles of Neural Networks for Speech Recognition, showing that inheriting teacher parameters provides a superior starting point in the parameter space, allowing the student to better capture the nuances of soft probability distributions.

This proximity drastically accelerates training. For example, in language model compression, Yihuan Mao et al. showed in LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression that initializing a student with the teacher's weights allows the model to reach target accuracy with an order of magnitude less training overhead compared to standard distillation from scratch.


Strategies for Structural Mismatches

Directly copying weights is straightforward when the student is an identical copy of the teacher. But how do we inherit weights when compressing a model into a smaller size or translating it into a completely different architecture?

Sub-network Extraction

When a student model is a shallower or narrower version of the teacher, researchers use layer-wise mapping or truncation strategies. In Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation, Cheng Chen et al. proposed recycling a subset of parameters from a large teacher to initialize custom student widths and depths, reducing total training operations by up to 70%.

Similarly, Chi Hu et al. found in The NiuTrans System for WNGT 2020 Efficiency Task that seeding a compact student decoder with the lower layers of a deep teacher model significantly enhances training stability. Beyond hidden layers, inheriting the final task logic is also critical; Hang Zhang et al. demonstrated in Intermediate Layer Distillation with the Reused Teacher Classifier that reusing the teacher's pre-trained classifier head allows the student to inherit task-specific logic directly, bridging the performance gap without requiring expensive pre-training.

Cross-Architecture Mapping

Weight inheritance is not limited to structural downscaling; it can also bridge fundamentally different mathematical operations. For instance, when distilling attention-based Transformers into linear-complexity State Space Models (SSMs) like Mamba, Junxiong Wang et al. showed in The Mamba in the Llama: Distilling and Accelerating Hybrid Models that pre-trained attention projection weights can be mapped to initialize the input and output projections of recurrent blocks. This cross-architecture warm start allows the recurrent student to retain the teacher's learned representations far better than training from scratch.

A similar principle applies when moving from graphs to vector spaces. In TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation, Ziang Zhou et al. directly transplanted weights from a teacher Graph Neural Network's feature transformation layers into a student MLP's fully connected layers, leveraging the mathematical equivalence between the operations to provide a high-quality initialization shortcut.


Stabilizing Complex Learning Dynamics

Beyond pure acceleration, teacher initialization acts as a vital semantic anchor in complex, unstable training environments like generative modeling and data-free distillation.

  • Generative Alignment: In generative adversarial frameworks, distilling unconditional GANs often fails due to "output discrepancy"—where the student and teacher generate completely different images from identical noise vectors. Guodong Xu et al. resolved this in Mind the Gap in Distilling StyleGANs by forcing the student to inherit the teacher's mapping network weights, anchoring the semantic content while allowing the backbone to optimize freely.
  • Preventing Model Collapse: For text-to-image diffusion models, Axel Sauer et al. noted in Adversarial Diffusion Distillation that initializing the student with pre-trained foundation weights is a strict requirement to prevent catastrophic model collapse during adversarial distillation into rapid, single-step generation.
  • Data-Free Environments: When distilling models without access to the original training data, students must learn from synthetic or adversarial samples. Xinyin Ma et al. showed in Adversarial Self-Supervised Data-Free Distillation for Text Classification that transferring the bottom-layer weights of the teacher to the student provides a stable foundational alignment that mitigates the optimization gap caused by missing data distributions.

The Local Optima Trap

While weight inheritance is highly effective for model compression and architectural translation, it is not a universal silver bullet. The strategy relies on the assumption that the student should reside in the same representational neighborhood as the teacher.

When the goal shifts from model compression to model expansion—such as using a smaller teacher to jumpstart a larger student—this assumption can break down. As explored by the authors of A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models, forcing a larger student model to inherit the weights of a smaller teacher can bias the student too heavily toward the weaker model's representational space. This constraint can trap the larger model in local optima, preventing it from effectively utilizing its expanded capacity.

Consequently, teacher-guided initialization is most powerful when used as a architectural bridge or a compression shortcut, ensuring that a smaller or faster model can stand firmly on the functional shoulders of the giant that preceded it.

Key Foundational Work

Distilling Knowledge from an Ensemble of Models for Punctuation Prediction (2017) — contributed 2 concept elements to this direction.
Distilling Knowledge from Ensembles of Neural Networks for Speech Recognition (2016) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 90 papers
  2. Version 1 · 2026-10-04 · 160 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cross_model_weight_initialization_3134c472,
  title = {cross-model weight initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-model-weight-initialization/rdr_3134c472c7c944a98033451ebd8d8907},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_cross_model_weight_initialization_3134c472_v2,
  title = {cross-model weight initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-model-weight-initialization/rdr_3134c472c7c944a98033451ebd8d8907/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake