When training deep neural networks on a sequence of tasks or domains, standard practice often dictates initializing each new phase randomly or resetting to a static pre-trained checkpoint. However, this ignores a valuable asset: the geometric and semantic structure already carved into the weight space by preceding tasks.
Instead of starting from scratch, sequential weight initialization uses the optimized parameters of prior tasks to bootstrap the learning of new ones. This mechanism drives forward knowledge transfer, allowing models to converge faster, generalize better, and operate efficiently even when data for the new task is scarce.
The Intuition: Navigating the Loss Landscape
Why does initializing a new model or module with old weights work better than a random start? Consider the optimization landscape. A random initialization drops the model onto a high-altitude, chaotic plateau of the loss surface. From there, the optimizer must spend significant time and data discovering basic structural features—such as edge detectors in vision or syntactic patterns in language.
By contrast, sequential weight initialization acts as a warm restart. It places the model's starting point inside or near a well-optimized valley carved out by previous tasks. Because related tasks share underlying regularities, this initialization provides an advanced structural bias. As explored by Ryo Karakida et al. in Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting, utilizing the final parameters of a prior task as the starting point fundamentally alters how subsequent information is integrated, dictating whether prior knowledge accelerates or hinders new learning.
Direct Continuation and Domain Adaptation
The simplest form of this idea is a direct parameter handoff between sequential domains or stages. In gradual domain adaptation, where a model encounters a shifting data distribution, initializing the weights of the current domain with the optimized parameters of the immediate predecessor is crucial. Shogo Sagawa demonstrates in Cost-effective Framework for Gradual Domain Adaptation with Multifidelity that this continuous weight refinement bridges massive distribution gaps that would otherwise cause training to fail when labeled data is scarce.
Similarly, in graph learning, Lukas Galke Poech et al. show in Lifelong Learning in Evolving Graphs with Limited Labeled Data and Unseen Class Detection that warm restarts serve as an implicit knowledge transfer mechanism, buffering the network against catastrophic forgetting when adapting to emerging classes.
Cross-Stage Parameter Transfer
Sequential initialization is not limited to identical architectures processing sequential data streams; it also applies to multi-stage pipelines where different tasks share a core representation space.
For instance, in multi-hop question answering, Jinhao Jiang et al. design UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph to initialize a reasoning model using the weights of a previously trained retrieval model. This ensures that the fine-grained reasoning phase inherits the relational mappings and semantic representations established during the initial search phase.
In computer vision, Sijie Zhu applies a similar logic in BDANet: Multiscale Convolutional Neural Network with Cross-directional Attention for Building Damage Assessment from Satellite Images, initializing a complex classification task with weights learned from a prior localization task to exploit shared spatial features and accelerate convergence.
Parameter Aggregation and Composition
When a model encounters a long sequence of tasks, initializing solely from the immediately preceding task might cause it to overfit to the recent past or lose broader context. To counter this, advanced strategies aggregate parameters from multiple historical states.
One approach is parameter averaging. In BiKT: Enabling Bidirectional Knowledge Transfer Between Pretrained Models and Sequential Downstream Tasks, Chaoyue Niu et al. initialize task-specific parameters by averaging learned parameters from all previous tasks, providing a robust starting point in low-data regimes.
A more sophisticated approach scales or selects parameters based on their relevance. Fuli Qiao et al. in Learn more, but bother less: parameter efficient continual learning compute the sensitivity of existing parameter triplets using a first-order Taylor expansion. By initializing new parameters with a weighted combination of the most sensitive (and thus informative) components from previous tasks, the model maximizes forward transfer and accelerates convergence without relying on naive baselines.
Bootstrapping Modular and Subnetwork Architectures
To completely eliminate catastrophic forgetting, many continual learning frameworks freeze historical parameters and allocate new, lightweight modules (like adapters or calibration blocks) for new tasks. Sequential weight initialization plays a vital role here by ensuring these new modules do not start blind.
In Calibrating CNNs for Lifelong Learning (CCLL), Pravendra Singh et al. initialize the parameters of a new task's calibration modules with the weights learned from the preceding task, allowing the model to leverage accumulated representations. Similarly, Lixin Su et al. show in Continual Domain Adaptation for Machine Reading Comprehension that initializing new domain-specific modules with weights from a previously mastered domain allows the model to inherit relevant linguistic features.
Even within a single dense network, strategies like sequential weight rewinding—used by Tianlong Chen 0001 et al. in Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning—initialize sparse subnetworks for new tasks using weights optimized during previous tasks, proving that the structural advantages of prior learning can be preserved and reused indefinitely.
Key Foundational Work
Beating a Defender in Robotic Soccer: Memory-Based Learning of a Continuous Function (1995) — contributed 2 concept elements to this direction.