Pseudo-Labeling for Knowledge Retention in Incremental Learning

This direction is merged into another direction: incremental pseudo-labeling. Its last version stays readable and citable here.

Version 1 of 156 papers68 conceptsfrozen 2026-10-04

When a neural network learns new tasks in sequence, it often suffers from "catastrophic forgetting"—the tendency to overwrite weights essential for old tasks to accommodate new ones. In structured prediction tasks like semantic segmentation or named entity recognition, this problem is exacerbated by background shift. When a model is trained on new categories, instances of previously learned categories often appear in the new images or text but are not labeled. Consequently, the model is explicitly taught to treat its old knowledge as "background" or "noise."

To combat this, a recurring strategy across recent research is the use of pseudo-labeling. By using a frozen snapshot of the model’s previous state to label new data, researchers provide a synthetic supervision signal that prevents the model from unlearning the past.

The Background Shift Problem

In incremental learning, the dataset for Task $B$ usually only contains ground-truth labels for new classes. However, objects from Task $A$ are often physically present in the background. As Chao Shang and colleagues point out in their work on Incrementer, this creates a "noisy background signal." The model sees a previously learned object (like a car) but is told the label is "background," forcing it to aggressively shift its decision boundaries.

Mining the Past with Teacher Models

The core intuition behind pseudo-labeling is to treat the previous iteration of the model as a "teacher" that knows what the background is hiding.

  1. Generating Synthetic Supervision: Before updating the weights, the current training batch is passed through the old model. This model generates "soft labels" or "pseudo-masks" for the classes it recognizes.
  2. Label Fusion: These pseudo-labels are merged with the new ground-truth labels. As described by Guanglei Yang in Uncertainty-Aware Contrastive Distillation, this creates a comprehensive semantic map where the model is supervised on both new and old classes simultaneously.
  3. Filtering for Quality: To prevent the model from learning from its own past mistakes, researchers like Fabio Cermelli and Arthur Douillard in CoMFormer employ confidence-based filtering. Only the most certain predictions from the old model are used as pseudo-labels, ensuring that the training signal remains clean.

Why It Works: Decision Boundary Stabilization

Pseudo-labeling works because it anchors the latent representation space. Instead of allowing the model to freely move its weights to minimize loss on new classes, pseudo-labels act as a regularizer.

In the context of speech recognition, Basil Abraham and Gurunath Reddy M demonstrate that using Kullback-Leibler (KL) divergence to align a new model's outputs with a teacher's soft labels maintains the decision boundaries of the original task. This "Learning Without Forgetting" (LwF) framework ensures that the model expands its knowledge rather than replacing it.

Similarly, in Extensible Cross-Modal Hashing, Tian-Yi Chen uses self-distillation to ensure that new data categories are mapped into a hash space that is compatible with old ones. By using the existing model to generate codes for new data, the system maintains "forward compatibility," eliminating the need to re-index the entire database.

Beyond Images: Replay and Unlabeled Data

The utility of pseudo-labeling extends to scenarios where historical data is entirely unavailable or restricted by privacy.

  • Unlabeled In-the-wild Data: Na Dong and Gim Hee Lee showed that one can use random, unlabeled images from the web as a proxy for the original training set. By pseudo-labeling these "in-the-wild" images with the old model, the network can practice its old skills on new, diverse examples, effectively bridging the gap between non-co-occurring classes.
  • Memory Rebalancing: In multi-label settings, Kaile Du and Fan Lyu introduced RebLL, which uses "Online Relabeling." This technique uses both the old and new models to fill in missing labels in a small memory buffer (replay buffer), ensuring the model doesn't become biased toward the most recently learned classes.

The Symbiosis of Semi-Supervised and Continual Learning

A particularly powerful evolution of this idea is the integration of semi-supervised learning. In SFNET, Yongrui Chen and Guilin Qi argue that these two fields are naturally symbiotic. Historical knowledge improves the accuracy of pseudo-labels for new, unlabeled tasks, while those same generated labels provide the diverse training signals necessary to prevent the forgetting of the past. This creates a virtuous cycle where the model becomes more robust as it encounters more data, even if that data is largely unlabeled.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_pseudo_labeling_for_knowledge_retention__6c0d4abb,
  title = {Pseudo-Labeling for Knowledge Retention in Incremental Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pseudo-labeling-for-knowledge-retention-in-incremental-learning/rdr_6c0d4abb628b477398b039cad0d8b90e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_pseudo_labeling_for_knowledge_retention__6c0d4abb_v1,
  title = {Pseudo-Labeling for Knowledge Retention in Incremental Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pseudo-labeling-for-knowledge-retention-in-incremental-learning/rdr_6c0d4abb628b477398b039cad0d8b90e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}