When we train a neural network from scratch, the optimization landscape behaves like a rugged mountain range. Two independent training runs, even with the same data and architecture, will settle into completely different, isolated valleys. If you try to average their weights, you end up on a high-error mountain peak between them. The resulting averaged model performs no better than random guessing.
However, the modern pre-training and fine-tuning paradigm fundamentally changes this geometry. When multiple models are initialized from the same pre-trained checkpoint, they no longer settle into isolated valleys. Instead, they navigate within the same broad, low-loss basin.
This geometric phenomenon is known as Linear Mode Connectivity (LMC). Because these models share a common basin, we can linearly interpolate or average their parameters directly in weight space without crossing high-error barriers. This unlocks a powerful capability: we can manipulate, merge, and improve models using simple parameter arithmetic, achieving the benefits of an ensemble at the computational cost of a single model.
Why Shared Initialization Anchors the Landscape
To understand why linear mode connectivity occurs, we must look at what happens during the pre-training phase. In What is being transferred in transfer learning?, Behnam Neyshabur et al. demonstrated that pre-training does more than just learn reusable features; it restricts the optimization process to a specific, highly stable region of the loss landscape.
Random Initialization:
[Basin A: Run 1] ---> /\/\ (High Error Barrier) <--- [Basin B: Run 2]
Shared Pre-trained Initialization:
[Pre-trained Anchor]
/ \
v v
[Fine-tuned Task 1] <---(Low-Loss Path)---> [Fine-tuned Task 2]
When fine-tuning begins, the shared pre-trained weights act as a gravitational anchor. Even if we fine-tune different instances of the model on entirely different datasets or objectives, the parameter updates remain small enough that the models stay within this shared basin. As a result, the path between any two fine-tuned endpoints remains flat and low in error.
Furthermore, Zhanpeng Zhou et al. showed in On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm that extensive pre-training induces a linear relationship between a model's weights and its internal features. This cross-task linearity explains why simple weight arithmetic translates so cleanly into predictable behavioral changes in the model's outputs.
Moving Beyond Ensembles with Model Soups
Traditionally, to get the best performance out of a machine learning pipeline, practitioners train multiple models with different hyperparameters and ensemble their outputs at inference time. While effective, ensembling multiplies the computational cost and latency by the number of models in the ensemble.
LMC offers a "free lunch" alternative. In Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, Mitchell Wortsman et al. introduced the concept of "Model Soups." Instead of averaging the predictions of various fine-tuned models, they averaged the weights themselves.
Because the models reside in the same low-loss basin, calculating the numerical mean of their parameters yields a single model that sits closer to the center of the basin. This central solution typically exhibits flatter curvature, which translates directly to superior generalization and higher accuracy—all while maintaining the exact inference speed of a single model.
Building on this, Alexandre Ramé et al. developed Diverse Weight Averaging (DiWA) in Diverse Weight Averaging for Out-of-Distribution Generalization. They proved that by intentionally maximizing the functional diversity of independent training runs before averaging their weights, the final interpolated model becomes significantly more robust to out-of-distribution (OOD) data shifts.
Blending Capabilities via Task Arithmetic
Because weight space behaves linearly within a shared basin, we can treat parameter changes as "task vectors" that can be added or subtracted to compose new model behaviors.
Balancing Robustness and Accuracy
Fine-tuning a zero-shot foundation model (like CLIP) on a specific downstream task usually destroys its original out-of-distribution robustness. To solve this, Mitchell Wortsman et al. proposed WiSE-FT in Robust fine-tuning of zero-shot models. By linearly interpolating the weights of the initial zero-shot model and the fine-tuned model, they created a hybrid network that retains the high target accuracy of fine-tuning alongside the broad robustness of the original zero-shot model.
Composing Multi-Objective Models
This geometric property also allows us to navigate trade-offs between entirely different objectives at test time. In Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards, Alexandre Rame et al. demonstrated that if you fine-tune multiple language models on different human preference rewards, you can smoothly blend those behaviors simply by adjusting the interpolation weights. This allows users to dynamically traverse a Pareto-optimal front of model alignments without undergoing expensive multi-objective retraining.
Similarly, the authors of Composing Parameter-Efficient Modules with Arithmetic Operation showed that parameter-efficient fine-tuning (PEFT) modules, such as LoRA adapters, can be added or subtracted like building blocks. For instance, adding a "toxicology" task vector or subtracting a negative behavior vector allows for modular customization of large language models.
Mitigating Catastrophic Forgetting
Linear mode connectivity also provides a geometric solution to one of deep learning's oldest problems: catastrophic forgetting in sequential learning.
When a model learns tasks sequentially, standard stochastic gradient descent pushes the parameters out of the low-loss region of older tasks to satisfy the requirements of the new task. In Linear Mode Connectivity in Multitask and Continual Learning, Iman Mirzadeh et al. leveraged LMC to show that the optimal multitask solution (learning all tasks at once) and sequential solutions are connected by straight lines of low error.
By introducing optimization constraints that force sequential updates to remain within these shared, low-error valleys, algorithms can adapt to new data streams while geometrically anchoring the parameters to preserve performance on previously learned tasks.
Key Foundational Work
Linear Mode Connectivity in Multitask and Continual Learning (2020) — contributed 2 concept elements to this direction.
What is being transferred in transfer learning? (2020) — contributed 2 concept elements to this direction.