Teacher-Student Distillation for Selective Machine Unlearning

15 concepts12 papers1 proposal

Machine unlearning aims to remove the influence of specific data points or concepts from a trained model without the prohibitive cost of retraining from scratch. The central challenge is a delicate trade-off: the model must "forget" the target information entirely while "remembering" its general utility and performance on the remaining data.

Recent research has converged on knowledge distillation (KD) as a primary mechanism to navigate this trade-off. By treating the original, fully-trained model as a "teacher" and the unlearning model as a "student," researchers can use the teacher’s internal representations to guide the student in selectively discarding or preserving specific knowledge.

The Mechanism of Selective Disobedience

In standard knowledge distillation, a student model learns to mimic a teacher's output distribution to capture its expertise. In the context of unlearning, this relationship is inverted or bifurcated. As Kurmanji et al. describe in their SCRUB framework, the student is optimized to "selectively disobey" its original self.

This typically involves a dual-objective loss function:

  1. Retention: On the data that should be kept (the "retain set"), the student minimizes the divergence from the teacher's outputs to maintain accuracy.
  2. Erasure: On the data to be forgotten (the "forget set"), the student is encouraged to maximize divergence from the teacher or mimic a "noisy" or "uninformed" distribution.

Acharya and Aryal demonstrate that this dual distillation process is critical for Large Language Models (LLMs), where a negative loss function can suppress sensitive information while a standard distillation loss prevents the model from losing its linguistic capabilities.

Stabilizing Decision Boundaries

A major risk in unlearning is that updating weights to forget one sample might inadvertently distort the decision boundaries for unrelated samples. Kim et al. argue that knowledge distillation acts as a stabilizer. By forcing the student to match the teacher’s logits on non-target data, the unlearning process preserves the original decision boundaries, allowing the model to reach performance levels nearly identical to a model retrained from scratch.

This stabilizing effect is particularly useful in complex architectures. For instance, Sinha et al. adapted this approach for Graph Neural Networks (GNNs), using distillation to manage the structural dependencies between nodes and edges, ensuring that removing one node doesn't collapse the representation of its neighbors.

Unlearning Without the Retain Set

One of the most significant hurdles in machine unlearning is the requirement for the "retain set"—the massive original dataset that the model is supposed to keep. In many real-world scenarios, this data is no longer accessible due to privacy or storage constraints.

To solve this, Bonato et al. propose using Out-of-Distribution (OOD) images as "distillation anchors." Instead of needing the original training data, the student mimics the teacher's behavior on unrelated, surrogate images. This ensures the model's general decision logic remains intact while the specific target classes are scrubbed. Similarly, the CLUE algorithm (art_988a7b551cb54e16ab4fdc7e6602489f) applies this logic to mobile devices, where resource constraints make keeping the full retain set impossible.

Multi-Teacher Frameworks for Complex Tasks

As unlearning tasks become more complex—such as removing sequential concepts from diffusion models or user data from recommendation systems—researchers have moved toward multi-teacher architectures.

  • Recommendation Systems: Wang et al. propose E2URec, which uses two teachers: one to guide the model toward a "forgotten state" for specific user samples and another to ensure the preservation of general recommendation logic.
  • Generative Models: In the CCRT framework for diffusion models, Han et al. use distillation to remove artistic styles or NSFW content. The student mimics an unedited teacher for general prompts but follows a removal loss for target concepts, preventing "entity forgetting" where removing one concept (e.g., "Van Gogh") accidentally damages related concepts (e.g., "sunflowers").

Federated and Distributed Unlearning

The teacher-student paradigm is also central to Federated Unlearning (FU), where data is distributed across many clients. Mora et al. and Jeong and Ma highlight KD as a primary recovery mechanism in FU. Because retraining a global model across thousands of devices is impossible, a "virtual teacher" can be used locally on a client's device to penalize the influence of forgotten data while maintaining the global model's shared knowledge (Mora et al.).

By quantifying knowledge through internal representations or output logits, these distillation frameworks provide a mathematically grounded way to "edit" a model's memory, moving machine unlearning from a theoretical possibility to a scalable engineering tool.

Key Foundational Work

Towards Unbounded Machine Unlearning (2023) — contributed 2 concept elements to this direction.
Distill to Delete: Unlearning in Graph Networks with Knowledge Distillation (2023) — contributed 2 concept elements to this direction.
Is Retain Set All You Need in Machine Unlearning? Restoring Performance of Unlearned Models with Out-of-Distribution Images (2024) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_knowledge_distillation_for_unlearning_a1d9c1bf,
  title = {knowledge distillation for unlearning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/knowledge-distillation-for-unlearning/rdr_a1d9c1bf73a84ff1bd7ac53e17f7cf06},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_knowledge_distillation_for_unlearning_a1d9c1bf_v1,
  title = {knowledge distillation for unlearning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/knowledge-distillation-for-unlearning/rdr_a1d9c1bf73a84ff1bd7ac53e17f7cf06/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake