In the standard machine learning paradigm, training a model on a new task usually begins from one of two points: a randomly initialized state or a static, pre-trained base model. However, in "continual learning" scenarios—where tasks arrive in a sequence—this approach creates a fundamental tension. Starting from a generic base model ignores the specialized knowledge gained from previous tasks, while starting from the most recent "expert" can lead to "catastrophic forgetting" of earlier information.
A growing body of research suggests that the solution lies in sequential initialization and merging. By carefully choosing the starting weights for a new task and merging the resulting models back into a unified structure, researchers are finding they can achieve "forward transfer" (learning new tasks faster) and "backward transfer" (improving old tasks) simultaneously.
The Geometry of the Shared Basin
To understand why initialization matters, we have to look at the loss landscape of neural networks. If two models are trained on different tasks starting from the same base weights, they often end up in different "valleys" or basins of the loss landscape. Attempting to merge these models by averaging their weights usually results in a model that sits on the "ridge" between valleys, performing poorly on both tasks.
Hiroshi Kera and colleagues address this in Adapter Merging with Centroid Prototype Mapping for Scalable Class-Incremental Learning. They demonstrate that by initializing a new task's parameters with the weights learned from the initial task, rather than a generic pre-trained model, the parameters remain within a shared low-loss basin. This geometric alignment allows multiple specialized modules to be merged via simple weight averaging without losing accuracy.
Temporal Integration and Stability
When tasks arrive over a long timeline, the challenge shifts from managing pairs of tasks to managing a trajectory of knowledge. If you always initialize Task $N$ from Task $N-1$, the model might drift too far from the original distribution, a phenomenon sometimes called "concept drift" or "weight plasticization."
In How to Merge Your Multimodal Models Over Time?, Sebastian Dziadzio and his team introduced the TIME (Temporal Integration of Model Expertise) framework. They discovered that the specific mathematical merging technique is often less important than the initialization strategy. They found that initializing a new task using an Exponential Moving Average (EMA) of all previous experts provides a stable foundation. This "Best-in-TIME" approach ensures the model doesn't start from a blank slate or a single biased expert, but rather from the "center of gravity" of all cumulative knowledge.
Similarly, Imad Eddine Marouf argues in Weighted Ensemble Models Are Strong Continual Learners that maintaining a consistent trajectory in weight space is key. By initializing each new task with merged weights from all previous tasks, the model stays in a region where weight interpolation remains effective.
Selective Inheritance and Forward Transfer
Not all knowledge from the past is relevant to the future. Blindly inheriting all weights can sometimes introduce noise. Recent work has focused on "selective" initialization—identifying exactly which neurons or "synapses" should be reused.
Fuli Qiao explores this in Learn more, but bother less: parameter efficient continual learning. Instead of a simple average, the method calculates the "sensitivity" of previous parameters using a Taylor expansion. By initializing new task parameters as a weighted combination of the most relevant components from previous tasks, the model achieves significant "forward transfer," mastering new information much faster than it would from a random baseline.
This concept of "intelligent synapses," originally popularized by researchers like Friedemann Zenke in Intelligent synapses for multi-task and transfer learning, treats the internal structure of the network as an inductive bias. The model doesn't just remember data; it remembers the utility of its weights.
Warm Starts and Jumpstarts
The intuition behind these methods is often described as a "warm start." In Lifelong Domain Adaptation via Consolidated Internal Distribution, Mohammad Rostami describes a "jumpstart effect." Because the model begins a new task with a high baseline accuracy derived from previous domains, the adaptation process is significantly accelerated.
This is seen across diverse architectures:
- Reinforcement Learning: Qiyang Li and Sergey Levine show in Understanding the Complexity Gains of Single-Task RL with a Curriculum that initializing a policy with weights from a simpler preceding task can shift the computational complexity of a problem from exponential to polynomial.
- Prompt Tuning: In PCL: Prompt-based Continual Learning for User Modeling in Recommender Systems, Fan Yang demonstrates that initializing a new "prompt" with the optimized parameters of the previous task allows the model to inherit knowledge without the need for storing massive amounts of historical data.
By treating initialization not as a starting line, but as a handoff between experts, these architectures transform continual learning from a struggle against forgetting into a process of cumulative refinement.