Informed Weight Initialization for Incremental Learning

This direction is merged into another direction: background shift. Its last version stays readable and citable here.

Version 1 of 19 papers15 conceptsfrozen 2026-10-04

When a neural network learns new tasks sequentially, it often suffers from "catastrophic forgetting"—the tendency to overwrite old knowledge to make room for the new. A primary driver of this forgetting is the "cold start" problem: when we add a new classification head, we typically initialize its weights randomly. This creates a massive gradient mismatch between the well-tuned feature extractor and the chaotic new head, forcing the entire network to undergo drastic updates that erase previously learned representations.

Recent research suggests a more elegant solution: instead of starting from scratch, we should initialize new classes using the "latent knowledge" already hidden within the model’s existing parameters.

The Background as a Knowledge Reservoir

In semantic segmentation, the "background" class is often treated as a catch-all for anything not currently being tracked. However, Fabio Cermelli, Massimiliano Mancini, and their colleagues argue in Modeling the Background for Incremental Learning in Semantic Segmentation that the background is actually a rich source of information.

When a model is trained to recognize a set of classes, the objects it will learn in the future are currently labeled as background. Consequently, the feature extractor is already learning to represent these "unknown" objects, and the background classifier head is already learning to respond to them.

The researchers introduced MiB (Modeling the Background), which initializes the weights of new class classifiers using the parameters of the existing background classifier. By doing this, the new class "inherits" the feature alignment of the background. This prevents the initial training phase from being a chaotic reorganization and instead treats it as a refinement of existing boundaries.

Leveraging Semantic Proximity

The same principle applies to generative models. In the context of few-shot incremental image synthesis, Chaerin Kong and Nojun Kwak developed the ConPro framework in Conservative Generator, Progressive Discriminator: Coordination of Adversaries in Few-shot Incremental Image Synthesis.

Rather than using a generic background initialization, they propose selecting weights from the most semantically similar previously learned category. If a model already knows how to generate a "dog," and it is tasked with learning to generate a "wolf," starting with the "dog" weights provides a much more stable and accurate foundation than random noise.

By using the discriminator’s feature space to measure semantic distance, the model identifies which existing "knowledge" is most relevant to the new task. This targeted initialization acts as a form of "warm-start," allowing the generator to converge faster and with fewer examples, which is critical in few-shot scenarios.

Why It Works: Alignment Over Innovation

The intuition behind this research direction is that alignment is more important than innovation during the early stages of incremental learning.

  1. Gradient Stability: Random initialization creates large loss values, leading to aggressive weight updates that destroy the feature extractor's delicate balance. Informed initialization keeps the loss within a manageable range.
  2. Feature Reuse: The feature extractor has already spent thousands of iterations learning to distinguish shapes, textures, and edges. By initializing new heads from existing ones (like the background or a similar class), we ensure the new head is "looking" at the features the network already knows how to produce.
  3. Semantic Continuity: In incremental learning, the definition of "background" or "other" is fluid. By explicitly modeling this transition—treating a new class as a "breakout" from the background—we maintain a consistent internal logic for the model.

By treating new tasks as extensions of existing knowledge rather than entirely new problems, these methods transform incremental learning from a process of constant disruption into one of continuous refinement.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_informed_weight_initialization_for_incre_18b4b4af,
  title = {Informed Weight Initialization for Incremental Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/informed-weight-initialization-for-incremental-learning/rdr_18b4b4af46014feaaa7f99ef28a23e98},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_informed_weight_initialization_for_incre_18b4b4af_v1,
  title = {Informed Weight Initialization for Incremental Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/informed-weight-initialization-for-incremental-learning/rdr_18b4b4af46014feaaa7f99ef28a23e98/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}