Identity Preserving Initialization for Stable Model Adaptation

297 concepts199 papers2 proposals

When adapting a massive, pre-trained foundation model to a new task or modality, we face a fundamental dilemma. The pre-trained model represents a highly optimized, fragile configuration of parameters. If we inject new parameters—such as adapters, cross-attention layers, or side networks—and initialize them randomly, we introduce immediate, catastrophic noise into the network.

At "Step 0" of training, this random noise disrupts the model's carefully calibrated representations. The model experiences an immediate spike in loss and a severe drop in performance. The optimizer must then spend the early phases of fine-tuning simply trying to "unlearn" this initialization noise and recover the baseline performance before it can begin learning the target task.

Identity-preserving initialization solves this problem by ensuring that any newly added architectural components mathematically evaluate to an identity transformation at the start of training. By making new parameters functionally "invisible" at Step 0, the adapted model behaves exactly like the original pre-trained model, providing a perfectly stable baseline for optimization.


The Mathematics of Invisibility

To understand why this approach works, we can look at how new parameters are integrated into a network's forward pass. There are three primary mechanisms used to achieve an identity state at initialization: additive zero-initialization, multiplicative scaling, and gated routing.

1. Additive Invisibility (Zero-Initialization)

The most common way to introduce an adapter without altering baseline behavior is to parameterize the modification as an additive residual step. Consider a pre-trained weight matrix $W_0$ processing an input $x$. In Low-Rank Adaptation (LoRA), proposed by Hu et al. in Low-Rank Adaptation of Large Language Models, the forward pass is modified by adding a low-rank decomposition path:

$$h = W_0 x + \Delta W x = W_0 x + B A x$$

To ensure that $\Delta W = 0$ at Step 0, one of the two low-rank matrices ($A$) is initialized via a random Gaussian distribution, while the other ($B$) is initialized entirely to zero. Because $B = 0$, the product $BA$ is exactly zero. The entire auxiliary path vanishes, leaving only the original $W_0 x$ output.

This same principle is applied by Fantacci et al. in Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation, where structural adapters are integrated via skip-connections and forced to zero at the start of training to preserve existing representational knowledge.

2. Multiplicative Invisibility (Identity Mapping)

When an adapter acts directly on the activation path rather than as a parallel branch, initializing weights to zero would destroy the signal entirely. Instead, the parameters must be initialized to represent an identity matrix ($I$), where the output matches the input ($f(x) = x$).

In Advancing Parameter Efficiency in Fine-tuning via Representation Editing, Wu et al. introduce learnable scaling and bias vectors directly to hidden layers. By initializing the scaling factors to $1$ and the bias terms to $0$, the transformation begins as a pure identity function.

Similarly, Huang leverages this approach in TAda! Temporally-Adaptive Convolutions for Video Understanding by initializing an adaptive calibration function to output a vector of ones, ensuring that a temporally-extended convolution initially behaves exactly like a standard, pre-trained 2D spatial convolution.

3. Gated Invisibility

When integrating entirely new modalities—such as adding visual capabilities to an audio model—networks often use cross-attention layers. To prevent these new layers from destabilizing the model, authors frequently employ a gated routing mechanism:

$$h = h_{\text{original}} + \gamma \cdot \text{Attention}(h_{\text{original}}, x_{\text{new}})$$

By initializing the scalar gating coefficient $\gamma$ to zero, the network completely ignores the new sensory inputs at the start of training. Rouditchenko et al. utilize this strategy in Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation, allowing a frozen audio model to maintain its baseline performance perfectly until the optimizer gradually adjusts $\gamma$ to incorporate supplementary visual information.


Why It Works: Eliminating the Recovery Phase

The core intuition behind identity-preserving initialization lies in the optimization landscape. When a model is fine-tuned with randomly initialized adapters, the initial loss landscape is highly turbulent due to the injected noise.

Standard Initialization:
Step 0 (Random Noise) ──> Loss Spikes ──> Recovery Phase (Unlearning Noise) ──> Task Adaptation

Identity Initialization:
Step 0 (Identity State) ───────────────────────────────────────────────────────> Task Adaptation

As observed by Berto et al. in RouteFinder: Towards Foundation Models for Vehicle Routing Problems, zero-initializing adapter weights completely eliminates this initial performance drop. Because the model starts from a known, high-performing baseline, gradients flowing through the network are stable and meaningful from the very first iteration.

Furthermore, Zhao et al. demonstrate in Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters that starting from an identity state forces the model to learn task-specific "nudges" to the existing embeddings rather than completely reconstructing representations from scratch. This drastically reduces the parameter footprint required for successful adaptation.


Structural Scaling and Reversibility

The utility of identity-preserving initialization extends beyond parameter-efficient tuning blocks; it can be used to alter the macro-architecture of a model itself.

Vertical Layer Expansion

When scaling a model vertically by adding new layers, researchers often copy existing pre-trained layers to fill the new slots. However, simply adding duplicated layers alters the depth of the network and disrupts the residual stream.

To combat this, Tan et al. introduce DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs, where new layers are integrated by copying existing ones but zero-initializing their output projection weights. This ensures that the expanded model's initial behavior remains identical to the base model, providing a safe starting point for deeper fine-tuning. Zhang et al. apply a similar function-preserving initialization in ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning to scale architectures smoothly during continual pre-training.

Modifying Architecture Constraints

Sometimes, the goal is to change how a model processes data internally—for instance, converting a standard model into a memory-efficient, reversible network. Liao et al. demonstrate in Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning that the success of such structural transformations hinges entirely on preserving the model's original output behavior at initialization. By ensuring that the new reversible adaptation layers initially function as identity mappings, models can be transitioned into entirely new structural frameworks without losing their pre-trained capabilities.


Non-Zero Identity Exploration

While zero-initialization provides perfect stability, it introduces a subtle constraint: it forces parallel adapters to start from a single point in the parameter space where the weight increment is exactly zero. This can sometimes limit the optimizer's ability to explore alternative representations early in training.

To address this limitation, Zhu et al. propose RidgeLoRA: Matrix Ridge Enhanced Low-Rank Adaptation of Large Language Models. By configuring adapters in a series connection rather than a parallel branch, the authors show that it is possible to construct identity-preserving initializations using non-zero matrices. This allows the model to begin fine-tuning from the exact functional state of the pre-trained model while simultaneously unlocking a more flexible and expressive parameter space for optimization from day one.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 183 papers
  2. Version 1 · 2026-10-04 · 225 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_identity_preserving_initialization_11e0a76b,
  title = {identity-preserving initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/identity-preserving-initialization/rdr_11e0a76b0f274c308ba43d3cbaf066f4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_identity_preserving_initialization_11e0a76b_v2,
  title = {identity-preserving initialization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/identity-preserving-initialization/rdr_11e0a76b0f274c308ba43d3cbaf066f4/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake