Knowledge Distillation for Data Scarce Scenarios

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2145 papers170 conceptsfrozen 2026-10-04

Knowledge Distillation (KD) is traditionally framed as a way to compress a "cumbersome" teacher model into a "compact" student by having the student mimic the teacher’s output probabilities. In the presence of massive datasets, this works because the teacher’s "soft labels" reveal the dark knowledge of how it generalizes between classes.

However, this mechanism breaks down when data is scarce. In few-shot or low-resource settings, the student model often fails to reach the teacher's performance, not because the student lacks the capacity to learn, but because the training signal becomes too sparse to map the teacher's complex decision boundaries. Recent research has shifted from simply mimicking outputs to active strategies for densifying the transfer signal.

The Problem of Data Sparsity

Standard distillation assumes that the available data points are sufficient to "probe" the teacher's expertise. As Xiang Deng and Zhongfei Zhang argue in Personalized Education: Blind Knowledge Distillation, the performance gap between models is often caused by Blind Knowledge Regions (BKR)—areas of the input space where the student has never seen the teacher's logic.

When training data is limited, the student only sees the teacher's response at a few isolated coordinates. In binary or few-class classification, this is even worse: the output probabilities provide almost no information about the teacher's underlying feature representations. Noel Loo and colleagues show in Linear Projections of Teacher Embeddings for Few-Class Distillation that in these cases, the teacher's signal is too "thin" to guide the student effectively.

Densifying the Signal through Geometry

To overcome this, researchers have moved toward distilling the geometric and relational properties of the teacher's latent space.

Instead of looking at a single point, Locally Linear Region Knowledge Distillation forces the student to mimic the teacher across the entire linear region between data points. By interpolating between sparse samples, the student effectively "probes" the teacher's logic in the empty spaces of the training set.

Similarly, Pengfei Fang and Hui Xue propose Displacement Knowledge Distillation (DKD). Rather than just matching how a teacher sees an image, the student learns the vector displacement—the distance and direction—between different samples. This relational data is much richer than individual labels, providing a more robust anchor for the student when samples are few.

Expanding the Textbook

If the existing data is insufficient, another direction is to synthetically expand the "textbook" from which the student learns.

  1. Meta-Knowledge: Chengyu Wang and Yaliang Li developed Meta-KD, which uses a "meta-teacher" to capture transferable knowledge across multiple domains. This global expertise acts as an informative prior that supplements the limited local data available in a specific niche task.
  2. Model Inversion: When data is extremely scarce or entirely missing, Kuluhan Binici suggests using Condensed Sample-Guided Model Inversion. This technique uses a tiny set of "prototype" samples to guide the generation of synthetic data, ensuring the student sees a diverse but relevant distribution of inputs.
  3. Retrieval-Based Augmentation: In Rethinking Task-Specific Knowledge Distillation, Chang Liu and Chongyang Tao demonstrate that a "Contextualized Corpus" retrieved from a massive general dataset can serve as a better training ground than the small task-specific dataset alone.

Distillation as an Informative Prior

The unifying intuition across these papers is that in low-data regimes, the teacher model should not be treated as a simple labeler, but as a source of structural priors.

As David Lopez-Paz and Leon Bottou noted in their work on Generalized Distillation, the guidance from a teacher model is most valuable when training data is at its minimum. By transferring "privileged information"—such as the teacher's internal rationales or its understanding of feature importance—distillation allows student models to generalize far beyond what the raw data would normally permit.

Papers in this version

Showing 60 of 145 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 171 papers
  2. Version 1 · 2026-10-04 · 145 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_data_efficient_knowledge_distillation_2b15fdcd,
  title = {data-efficient knowledge distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-efficient-knowledge-distillation/rdr_2b15fdcd1e28487a9a49910ce0138a4c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_knowledge_distillation_for_data_scarce_s_2b15fdcd_v1,
  title = {Knowledge Distillation for Data Scarce Scenarios},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/knowledge-distillation-for-data-scarce-scenarios/rdr_2b15fdcd1e28487a9a49910ce0138a4c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}