Logit Distillation for Model Alignment and Compression

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2814 papers1,059 conceptsfrozen 2026-10-04

In machine learning, a "hard label" is a blunt instrument. Telling a model that an image is simply a "dog" ignores the rich structural information the model has already learned—for instance, that the image also looks a bit like a "cat" but nothing like a "car."

Logit Distillation (or logit-based knowledge distillation) is the process of training a smaller "student" model to mimic the full probability distribution—the logits—of a larger "teacher" model. By matching these "soft labels," the student inherits the teacher's nuanced understanding of the relationships between classes, its uncertainty, and its decision-making logic.

The Intuition of Soft Labels

When a teacher model outputs a probability distribution, it reveals its internal "dark knowledge." If a teacher assigns a 90% probability to "Golden Retriever" and 9% to "Labrador," it is signaling to the student that these two categories are semantically related.

As Kalle Kujanpää and colleagues demonstrate in Knowledge Injection via Prompt Distillation, training on this full distribution is significantly more data-efficient than standard fine-tuning. It allows the student to learn the relative likelihood of various responses, preventing the model from simply mimicking surface-level style and instead forcing it to internalize factual nuances. This distributional mimicry is so effective that it can "bake" knowledge into a model's weights with 10x the efficiency of traditional methods.

Architectural Agnosticism

One of the primary advantages of logit distillation over "feature-based" distillation (which matches internal hidden states) is that it is architecturally agnostic. Because the distillation happens at the final output layer, the teacher and student do not need to share the same internal structure.

This flexibility allows researchers to bridge the gap between different model families:

Preventing Forgetting and Maintaining Loyalty

Logit distillation acts as a powerful anchor for a model's parameters. In Lifelong Pretraining, Xisen Jin and others use logit-based distillation to prevent "catastrophic forgetting." By requiring the updated model to match the output distributions of its previous iteration, the model retains old knowledge while adapting to new data streams.

This alignment goes beyond mere accuracy; it transfers the teacher's "personality." Utkarsh Ojha and Yuheng Li found that logit distillation causes students to inherit the teacher's specific failure modes, biases, and adversarial vulnerabilities. This "behavioral loyalty," as explored by Ke Xu in Beyond Preserved Accuracy, is critical for production environments where predictable confidence scores are just as important as correct labels.

Efficiency at Scale

While many complex variants of distillation exist, Zhiwei Hao argues in Revisit the Power of Vanilla Knowledge Distillation that "vanilla" logit matching—using simple Kullback-Leibler (KL) divergence—is often the most effective strategy at scale.

This simplicity enables massive compression:

  • Miniaturization: Henry Tsai and the Google team distilled BERT into a "MiniBERT" that is 27x faster on a CPU in Small and Practical BERT Models for Sequence Labeling.
  • Quantization: In LLM-QAT, Zechun Liu demonstrated that matching the full probability distribution allows models to be compressed to 4-bit precision without losing the nuanced decision boundaries of the original high-precision architecture.

By focusing on the distribution of possibilities rather than a single prediction, logit distillation provides a richer, more flexible supervisory signal that allows small models to punch far above their weight class.

Papers in this version

Showing 60 of 814 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 425 papers
  2. Version 1 · 2026-10-04 · 814 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_soft_label_distillation_2c5ce380,
  title = {soft label distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/soft-label-distillation/rdr_2c5ce380ae6048ed9bff6afd87174d32},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_logit_distillation_for_model_alignment_a_2c5ce380_v1,
  title = {Logit Distillation for Model Alignment and Compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/logit-distillation-for-model-alignment-and-compression/rdr_2c5ce380ae6048ed9bff6afd87174d32/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}