Linear Weight Interpolation in Parameter Space

405 concepts249 papers2 proposals

When fine-tuning a deep learning model on a specific target dataset, practitioners face a fundamental trade-off: specialization versus generalization. Standard fine-tuning optimizes a model for high accuracy on a target task, but this specialization often destroys the model's out-of-distribution (OOD) robustness, general-purpose reasoning, or performance on previously learned tasks—a phenomenon known as catastrophic forgetting.

Traditionally, balancing these trade-offs required complex regularization techniques during training or expensive multi-task data mixtures. However, a powerful alternative has emerged: linear weight interpolation. Instead of modifying the training objective, researchers can arithmetically blend the parameter weights of different models after training to create a hybrid model that inherits the best capabilities of its constituents.

The Core Mechanism: Moving Within the Loss Basin

Mathematically, given a pre-trained base model with weights $W_{\text{base}}$ and a fine-tuned variant with weights $W_{\text{fine}}$, linear weight interpolation constructs a new set of weights $W_{\text{interp}}$ via a simple convex combination:

$$W_{\text{interp}} = (1 - \alpha)W_{\text{base}} + \alpha W_{\text{fine}}$$

where $\alpha \in [0, 1]$ is a mixing coefficient that tunes the balance between the two models.

Intuitively, averaging the weights of two distinct neural networks should yield a broken model. Because neural networks are highly non-linear, a point halfway between two arbitrary local minima typically lands in a region of high loss.

Weight interpolation works because of a geometric property known as linear mode connectivity. When two models share the same initialization (i.e., they both start from the same pre-trained checkpoint $W_{\text{base}}$), their optimization trajectories do not diverge into isolated valleys. Instead, the fine-tuning process moves the parameters within a wide, flat low-loss basin. Because this region is convex, the straight line segment connecting $W_{\text{base}}$ and $W_{\text{fine}}$ remains entirely within the low-loss plateau.

As a result, sampling a point along this line allows the model to retain the structural representations of the base model while incorporating the specialized features of the fine-tuned model.

   Loss Landscape

   High Loss (Mountain)
        /\
       /  \      Linear Interpolation Path
      /    \    [========================]
     /      \  W_base                W_fine
____/        \______________________________/
               Low-Loss Basin (Shared Plateau)

In Robust fine-tuning of zero-shot models, Mitchell Wortsman demonstrated that this simple interpolation technique (termed WiSE-FT) creates a single set of weights that achieves high accuracy on the target task while preserving or even enhancing the model's broad robustness to distribution shifts.

Building on this intuition, Ilharco et al. introduced PAINT in Patching open-vocabulary models by interpolating weights. They showed that weight interpolation can be used to "patch" specific weaknesses in open-vocabulary models like CLIP. By varying $\alpha$, practitioners can trace a continuous Pareto frontier of accuracy trade-offs, recovering general robustness just as effectively as early stopping or complex optimization constraints, but entirely post-hoc.

Mitigating Forgetting and Regression Errors

The stability offered by shared loss basins makes weight interpolation an effective tool for updating models without introducing regressions.

When a model is updated with new data, it often suffers from "negative flips"—instances where a previously correct prediction becomes incorrect after the update. In Backward Compatibility During Data Updates by Weight Interpolation, Schumann et al. showed that interpolating between legacy and updated weights smooths out decision boundaries, drastically reducing these regression errors without sacrificing the new model's overall performance gains.

This mechanism extends naturally to continual learning and robotics. For instance, robotic foundation models often lose their broad visual intelligence when fine-tuned on specific motor datasets. To counteract this, Zaech et al. proposed a gradual weight reversion schedule in ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models, linearly interpolating fine-tuned vision backbones back toward their original pre-trained states. This allows the model's reasoning layers to adapt to high-quality features while recovering broad semantic knowledge. Similarly, Roth et al. demonstrated in A Practitioner's Guide to Continual Multimodal Pretraining that post-training weight averaging serves as a highly stable baseline for balancing new domain knowledge with long-term zero-shot capabilities.

Blending Behavioral Styles and Personalization

Weight interpolation is not limited to balancing base models with specialized ones; it can also blend entirely different behavioral styles or user preferences.

In The Thinking Spectrum: An Empirical Study of Tunable Reasoning in LLMs through Model Merging, Lan et al. discovered that large language models fine-tuned for distinct behavioral styles—such as direct instruction-following versus slow, chain-of-thought (CoT) reasoning—remain structurally compatible in weight space. By arithmetically interpolating their parameters, they created a continuous "Thinking Spectrum." This allows developers to dynamically adjust the trade-off between logical accuracy and token efficiency at inference time without additional training.

Direct-Response Model (W_direct) <-------------------> Chain-of-Thought Model (W_CoT)
                                    Interpolated
                                 Thinking Spectrum

For user personalization, Jang et al. developed Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. Instead of training a single model to satisfy an "average" human preference, they trained separate models on distinct traits and performed a weighted linear interpolation of their parameters at inference time, allowing for scalable, on-the-fly personalization to match specific user profiles.

Preserving Geometry: Spherical Linear Interpolation (SLERP)

While simple linear interpolation works well for models that have not drifted far from their initialization, straight lines in high-dimensional vector spaces can introduce geometric distortions when merging highly divergent models.

In a high-dimensional parameter space, weight vectors often have a specific magnitude (length). If you interpolate along a straight line between two normalized vectors on a hypersphere, the intermediate points will cut through the interior of the sphere. This reduces the magnitude of the intermediate weight vectors, which can destabilize the internal representations of layers like attention projections or token embeddings.

        Linear Path (Cuts through sphere, changes magnitude)
         W_1 --------x-------- W_2
            \        |        /
             \       |       /
              \      |      /
               \     |     /
                \    |    /
                 \   |   /
                  \  |  /
                   \ | /
                    \|/
                   Origin

        Spherical Path (SLERP - Preserves magnitude)
               .--- x ---.
             /             \
            W_1             W_2

To preserve the geometric properties of the weight space, researchers frequently utilize Spherical Linear Interpolation (SLERP). SLERP interpolates along the spherical surface rather than a straight line, maintaining a constant vector magnitude throughout the transition.

This geometric preservation proves crucial when combining models with significantly different optimization paths. For example, Boros et al. applied spherical interpolation in Fine-Tuning and Retrieval Augmented Generation for Question Answering Using Affordable Large Language Models to successfully merge a fine-tuned task model with a Direct Preference Optimization (DPO) variant. Similarly, Corbeil et al. leveraged SLERP in A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment to merge specialized clinical expert models back into a general-purpose base model, successfully integrating domain expertise while preventing the loss of general reasoning skills.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 247 papers
  2. Version 1 · 2026-10-04 · 375 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_linear_weight_interpolation_weight_space_1423baec,
  title = {linear weight interpolation (weight-space interpolation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-weight-interpolation-weight-space-interpolation/rdr_1423baecfc954fe6a057faaee8df31b9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_linear_weight_interpolation_weight_space_1423baec_v2,
  title = {linear weight interpolation (weight-space interpolation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-weight-interpolation-weight-space-interpolation/rdr_1423baecfc954fe6a057faaee8df31b9/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake