Knowledge Distillation in Low Data Regimes

222 concepts166 papers1 proposal

Standard knowledge distillation (KD) assumes an abundance of data. To copy a teacher's complex decision boundary, a student model typically needs to sample that boundary extensively using a massive transfer dataset. When data is scarce, standard KD struggles because a few sparse data points cannot adequately chart the teacher's high-dimensional landscape, leading to a student model that overfits or fails to generalize.

However, recent insights show that KD is uniquely powerful precisely when data is limited, acting as a strong regularizer and an informative prior. As shown by Felix Sarnthein et al. in Revisiting Knowledge Distillation: The Hidden Role of Dataset Size, the relative performance gains of distillation actually widen as the available training data decreases. To achieve this data efficiency, researchers focus on a core principle: increasing the information density of the supervision signal so the student can extract more knowledge from fewer examples.

Dense Supervision via Internal Representations

If final output logits provide only a few bits of semantic information per sample, looking inside the teacher's intermediate layers provides a continuous stream of structural guidance.

In Knowledge Distillation from Internal Representations, the authors propose Stacked Internal Distillation (SID), forcing the student to match intermediate attention patterns and hidden states. This dense internal supervision allows the student to directly inherit the teacher's feature-processing pipeline without needing millions of examples to infer it implicitly.

Similarly, Siyue Wu et al. introduce Attribution-Driven Knowledge Distillation (AD-KD) in AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression. By transferring token-level attribution scores (the "rationale" behind a prediction calculated via integrated gradients), the teacher explicitly tells the student which features matter. This targeted hint compensates for sparse training signals by preventing the student from wasting capacity on irrelevant correlations.

The Informational Leverage of Soft Labels

We often think data efficiency requires synthesizing complex new inputs. However, Tian Qin et al. demonstrate in A Label is Worth A Thousand Images in Dataset Distillation that the true leverage point in low-data regimes is the soft label.

By pairing random real images with diffuse probability distributions from an expert model, the student absorbs "dark knowledge"—the relative correlations and geometric distances between classes. Interestingly, the authors find that labels from early-stopped, less-mature expert models are often more effective teachers because their softer, more diffuse distributions are easier for a data-constrained student to digest than the overconfident peaks of a fully converged model.

Causal and Structural Constraints

Another way to bypass the data bottleneck is to constrain the student's learning space, ensuring it models the underlying system dynamics rather than just mimicking static outputs.

Zhengxuan Wu et al. pursue this via Causal Distillation for Language Models, introducing the Distillation Interchange Intervention Training Objective (DIITO). Instead of forcing the student to copy static states, DIITO forces the student to mimic the causal dynamics of the teacher when internal states are counterfactually manipulated. Teaching functional relationships rather than raw outputs provides a structurally robust signal that generalizes well even when training data is highly limited.

In text generation, Tianyi Zhang et al. show in TempLM: Distilling Language Models into Template-Based Generators that distilling a language model into structured templates compensates for small datasets. Bounding the output space with rigid template constraints prevents the model from hallucinating in out-of-domain scenarios, proving that structural constraints can effectively substitute for large-scale training corpora.

Distributional Expansion

When data is limited to just a few shots, the student risks overfitting to the specific sample distribution of the transfer set. To counter this, distillation frameworks can explicitly model the uncertainty of the data space.

Shumin Kong et al. address this in Learning Student Networks with Few Data by utilizing Wasserstein distance to model a "neighborhood" of potential data distributions around the few available samples. This distributionally robust objective acts as a versatile regularizer, ensuring that the student generalizes smoothly across the broader data manifold even when the physical transfer set is extremely small.

Key Foundational Work

In Teacher We Trust: Learning Compressed Models for Pedestrian Detection (2016) — contributed 3 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 171 papers
  2. Version 1 · 2026-10-04 · 145 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_data_efficient_knowledge_distillation_2b15fdcd,
  title = {data-efficient knowledge distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-efficient-knowledge-distillation/rdr_2b15fdcd1e28487a9a49910ce0138a4c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 2, frozen text):

@misc{lacuna_data_efficient_knowledge_distillation_2b15fdcd_v2,
  title = {data-efficient knowledge distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-efficient-knowledge-distillation/rdr_2b15fdcd1e28487a9a49910ce0138a4c/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake