Dual Teacher Distillation for Stability and Plasticity

This direction is merged into another direction: dual-teacher distillation. Its last version stays readable and citable here.

Version 1 of 132 papers40 conceptsfrozen 2026-10-04

In machine learning, models often face a fundamental conflict known as the stability-plasticity dilemma. To learn something new (plasticity), a model must change its weights; however, changing those weights often causes it to overwrite what it already knows (stability). This leads to "catastrophic forgetting," where a model's performance on old tasks collapses as soon as it encounters a new one.

While standard Knowledge Distillation (KD) uses a single "teacher" to guide a "student," a growing body of research suggests that a single teacher is insufficient for sequential or distributed learning. Instead, researchers are increasingly using Dual-Teacher Distillation, where a student model is simultaneously anchored by two specialized teachers: one that preserves historical or global knowledge and another that masters the current task.

The Failure of the Single Teacher

In traditional Class-Incremental Learning (CIL), a model is often distilled from its own previous version to prevent forgetting. However, Changhong Zhong and Wei-Shi Zheng point out in Discriminative Distillation to Reduce Class Confusion in Continual Learning that this creates a "class confusion" problem. The old model knows the old classes well but knows nothing about the new ones. If the student only listens to the old model, it struggles to draw clear boundaries between visually similar old and new categories.

Similarly, in Federated Learning (FL), a central model learning from local devices can suffer from "edge bias." Sangho Lee and Nojun Kwak argue in Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation that if a model only distills from a local device's data, it quickly overfits to that device's specific biases, losing the generalized knowledge it gained from the rest of the network.

The Dual-Teacher Architecture

The dual-teacher approach solves this by providing two distinct signals that the student must satisfy simultaneously. While the specific implementation varies, the teachers generally fall into two functional roles:

  1. The Historian (Stability): This teacher is typically a frozen snapshot of the model from a previous state or a pre-trained foundation. Its job is to say, "Don't forget the general features and old classes we've already mastered."
  2. The Specialist (Plasticity): This teacher is often an "interim expert" trained specifically on the current batch of new data. Its job is to say, "Here is exactly how to distinguish these new features."

By minimizing the loss relative to both, the student finds a weight configuration that sits at the intersection of old expertise and new mastery.

Balancing Past and Future

In Bridge Past and Future: Overcoming Information Asymmetry in Incremental Object Detection, Qijie Mo and colleagues use this to solve the "background" problem in object detection. When a model learns new objects, it often treats old objects as "background." Their dual-teacher setup uses a previous-stage teacher to protect old class expertise while an interim expert model, trained only on current data, ensures the student learns the new objects with high precision.

Preserving Foundation Model "Magic"

A specialized version of this appears in Vision-Language Models (VLMs) like CLIP. Hongsheng Zhang and Zhong Ji introduce MulKI in Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning, which uses one teacher to represent the original pre-trained zero-shot capabilities and another to represent the most recent task state. This prevents the model from becoming a narrow specialist and losing its broad, general-purpose reasoning. Yu-Chu Yu further refined this in Select and Distill, using the disagreement between these two teachers to decide exactly which knowledge is most critical to transfer.

Application in Distributed and Federated Settings

In decentralized environments, the dual-teacher approach acts as a stabilizer. Anastasiia Usmanova and François Portet proposed FLwF-2T in A distillation-based approach integrating continual learning and federated learning for pervasive services. Here, the two teachers are the client's own previous local model (preserving local history) and the global server model (integrating network-wide knowledge).

This structure is also effective for semi-supervised learning in sensitive domains like medicine. Huisi Wu and Yuzhi Liu developed FedCD in Federated Semi-Supervised Learning with Class Awareness Balance via Dual Teachers, which uses a global teacher from the server and a local moving-average teacher. This ensures that hospitals can learn from their own unlabeled data without losing the collective patterns discovered by other institutions.

Why It Works: The Geometry of Knowledge

The intuition behind dual-teacher distillation is that it constrains the optimization path of the student. In a single-teacher setup, the student's gradients are pulled toward a single point in the loss landscape, which might be far from the "knowledge" of previous tasks.

By adding a second teacher, the student is effectively "tethered" to two points. Yoojin Choi and Mostafa El‐Khamy demonstrate in Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay that this dual-expert approach prevents the student from becoming biased toward new tasks.

Furthermore, Behrad Moniri and Hamed Hassani provide a theoretical perspective in On the Mechanisms of Weak-to-Strong Generalization, suggesting that strong models can learn "easy" task-specific features from a teacher while retaining "hard" general features from their own pre-training. Dual distillation formalizes this by explicitly providing both types of features as supervision signals, allowing the student to eventually outperform either teacher individually.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dual_teacher_distillation_for_stability__0c84c048,
  title = {Dual Teacher Distillation for Stability and Plasticity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dual-teacher-distillation-for-stability-and-plasticity/rdr_0c84c04826e7446cb464fadf074269ba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dual_teacher_distillation_for_stability__0c84c048_v1,
  title = {Dual Teacher Distillation for Stability and Plasticity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dual-teacher-distillation-for-stability-and-plasticity/rdr_0c84c04826e7446cb464fadf074269ba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}