Zero-Initialized Adaptation for Stable Model Fine-Tuning

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2225 papers278 conceptsfrozen 2026-10-04

When adapting a massive pre-trained model to a new task, the first few steps of training are often the most precarious. Standard practice involves adding new parameters—such as adapters, prefix tokens, or sidecar networks—and initializing them randomly. However, this random noise acts as a "shock" to the system, forcing the model to immediately deviate from its carefully learned pre-trained representations.

A growing body of research suggests a more elegant solution: Zero-Initialized Adaptation. By ensuring that new parameters contribute exactly zero to the model's output at the start of training, researchers can preserve pre-trained knowledge and achieve significantly more stable convergence.

The Identity Mapping Intuition

The core philosophy behind zero-initialization is the identity mapping. If a pre-trained model $f(x)$ is already highly capable, any adaptation module $A(x)$ should ideally start by doing nothing. Mathematically, we want the adapted model's output at step zero to be:

$$Output = f(x) + A(x) = f(x) + 0$$

By initializing the weights of $A(x)$ such that its output is zero, we ensure the fine-tuning process begins from the exact performance level of the pre-trained backbone. This prevents the "initial performance dip" common in few-shot learning and protects the model from catastrophic forgetting.

Zero-Initialized Gating and Attention

One of the most prominent applications of this idea is in Large Language Models (LLMs). In LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention, Dongyang Liu, Renrui Zhang, and their colleagues introduced a "zero-initialized attention" mechanism. Instead of letting new instruction-following prompts immediately influence the self-attention layers, they use a learnable gating factor initialized to zero.

As training progresses, the gate slowly opens, allowing the model to gradually integrate task-specific information without disrupting its foundational reasoning capabilities. Zhang later demonstrated in LLaMA-Adapter: Efficient Fine-tuning of Large Language Models with Zero-initialized Attention that this strategy allows billion-parameter models to be tuned in under an hour on a single node.

Preserving Generative Quality in Diffusion

The stakes for initialization are even higher in generative modeling, where random noise can instantly destroy the structural integrity of generated images. In the GWIT framework, Eleonora Lopez and colleagues use zero-initialized 1x1 convolutions to connect EEG-based conditioning to a Latent Diffusion Model. This ensures the model begins with its original generative capabilities intact, only learning to "steer" the generation as the adapter matures.

Similarly, Fanghua Yu and the team behind UniCon utilize zero-initialized transformation modules to siphon features from a frozen base model into a trainable adapter. By starting with a "closed" connection, the adapter begins training as a standard diffusion model, ensuring a stable transition as it learns to incorporate external guidance signals.

Variations in Low-Rank Adaptation

While the most common approach is to initialize a final projection layer to zero, researchers have explored several variations to balance stability with the ability to explore the parameter space:

  • Low-Rank Visual Prompting (LoR-VP): Can Jin and Ying Li apply this to vision models by initializing one matrix of a low-rank pair to zero and the other to random noise. This ensures a neutral initial effect while providing a gradient path for the non-zero components.
  • Multi-Task Democratization: In MultiLoRA, Yu Lin uses zero-initialized scaling factors to manage multiple parallel adapters, allowing the model to selectively "turn on" different task-specific modules.
  • Identity Scaling: Methods like HyperAdapt take a different route by initializing scaling factors to one (identity) rather than zero, which is effective when the adaptation is multiplicative rather than additive.

Why It Works: The Optimization Perspective

From an optimization standpoint, zero-initialization places the starting point of the fine-tuning trajectory exactly at the local minimum found during pre-training. As Claudio Fantacci and Mohit Sharma argue in their work on Lossless Adaptation, this "lossless" start means the optimizer doesn't have to spend the first few hundred iterations recovering from the damage caused by random weights.

This is particularly critical in domains with limited data. Chuan Zhou and Chunyuan Zheng found that partial zero-initialization significantly mitigates overfitting when fine-tuning on small, high-quality datasets, as it prevents the model from diverging drastically before it has seen enough target-domain evidence.

Papers in this version

Showing 60 of 225 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 183 papers
  2. Version 1 · 2026-10-04 · 225 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_identity_preserving_initialization_11e0a76b,
  title = {identity-preserving initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/identity-preserving-initialization/rdr_11e0a76b0f274c308ba43d3cbaf066f4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_zero_initialized_adaptation_for_stable_m_11e0a76b_v1,
  title = {Zero-Initialized Adaptation for Stable Model Fine-Tuning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-initialized-adaptation-for-stable-model-fine-tuning/rdr_11e0a76b0f274c308ba43d3cbaf066f4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}