In machine learning, a "hard label" is a blunt instrument. Telling a model that an image is simply a "dog" ignores the rich structural information the model has already learned—for instance, that the image also looks a bit like a "cat" but nothing like a "car."
Logit Distillation (or logit-based knowledge distillation) is the process of training a smaller "student" model to mimic the full probability distribution—the logits—of a larger "teacher" model. By matching these "soft labels," the student inherits the teacher's nuanced understanding of the relationships between classes, its uncertainty, and its decision-making logic.
The Intuition of Soft Labels
When a teacher model outputs a probability distribution, it reveals its internal "dark knowledge." If a teacher assigns a 90% probability to "Golden Retriever" and 9% to "Labrador," it is signaling to the student that these two categories are semantically related.
As Kalle Kujanpää and colleagues demonstrate in Knowledge Injection via Prompt Distillation, training on this full distribution is significantly more data-efficient than standard fine-tuning. It allows the student to learn the relative likelihood of various responses, preventing the model from simply mimicking surface-level style and instead forcing it to internalize factual nuances. This distributional mimicry is so effective that it can "bake" knowledge into a model's weights with 10x the efficiency of traditional methods.
Architectural Agnosticism
One of the primary advantages of logit distillation over "feature-based" distillation (which matches internal hidden states) is that it is architecturally agnostic. Because the distillation happens at the final output layer, the teacher and student do not need to share the same internal structure.
This flexibility allows researchers to bridge the gap between different model families:
- Cross-Family Transfer: Rui Yu used logit-space alignment to distill global contextual knowledge from a high-accuracy Transformer into a high-efficiency Mamba model for 3D detection, achieving a 2x speedup in Unleashing the Potential of Mamba.
- Ensemble Compression: Ruijie Yang showed that distilling the collective "wisdom" of multiple architectures (CNNs and Transformers) into a single student creates more generalized decision boundaries in Common Knowledge Learning for Generating Transferable Adversarial Examples.
Preventing Forgetting and Maintaining Loyalty
Logit distillation acts as a powerful anchor for a model's parameters. In Lifelong Pretraining, Xisen Jin and others use logit-based distillation to prevent "catastrophic forgetting." By requiring the updated model to match the output distributions of its previous iteration, the model retains old knowledge while adapting to new data streams.
This alignment goes beyond mere accuracy; it transfers the teacher's "personality." Utkarsh Ojha and Yuheng Li found that logit distillation causes students to inherit the teacher's specific failure modes, biases, and adversarial vulnerabilities. This "behavioral loyalty," as explored by Ke Xu in Beyond Preserved Accuracy, is critical for production environments where predictable confidence scores are just as important as correct labels.
Efficiency at Scale
While many complex variants of distillation exist, Zhiwei Hao argues in Revisit the Power of Vanilla Knowledge Distillation that "vanilla" logit matching—using simple Kullback-Leibler (KL) divergence—is often the most effective strategy at scale.
This simplicity enables massive compression:
- Miniaturization: Henry Tsai and the Google team distilled BERT into a "MiniBERT" that is 27x faster on a CPU in Small and Practical BERT Models for Sequence Labeling.
- Quantization: In LLM-QAT, Zechun Liu demonstrated that matching the full probability distribution allows models to be compressed to 4-bit precision without losing the nuanced decision boundaries of the original high-precision architecture.
By focusing on the distribution of possibilities rather than a single prediction, logit distillation provides a richer, more flexible supervisory signal that allows small models to punch far above their weight class.