Knowledge Distillation (KD) is the standard technique for training efficient student models, particularly in Non-Autoregressive Translation (NAT). By training a student to mimic the output distribution of a large teacher model, we simplify the mapping between source and target languages. However, research by Jannis Vamvas and Rico Sennrich suggests that this process acts as a "low-pass filter," systematically scrubbing away linguistic diversity and reinforcing majority-class biases.
The Mechanism of Lexical Erosion
The core utility of KD lies in its ability to reduce "multi-modality"—the phenomenon where a single sentence has many valid translations. The teacher model provides a smoother, more deterministic signal than raw human-authored data. While this makes the student's task easier, it comes at a high cost: the teacher tends to produce high-probability, simplified outputs.
As noted by Liang Ding and Longyue Wang, this process effectively washes out the "long tail" of language. Rare word senses, specialized terminology, and nuanced syntactic structures are replaced by common synonyms and frequent patterns. The student model doesn't just learn to translate; it learns a simplified, "distilled" version of the language that lacks the expressive range of the original data.
Why Distillation Amplifies Bias
When a teacher model encounters an ambiguous source sentence, its probability distribution often favors the most frequent association found in its training set. Vamvas and Sennrich demonstrate that KD amplifies these existing biases. Because the student is trained to maximize the likelihood of the teacher's most probable output, it becomes even more prone to overgeneralization than a model trained on raw data. This creates a feedback loop where minority linguistic forms and rare word senses are systematically erased in favor of majority-class patterns.
Rejuvenating the Long Tail
To counter this loss of information, researchers have proposed methods to "re-inject" the diversity of raw data into the training process.
- Low-Frequency Rejuvenation (LFR): Liang Ding and colleagues introduced strategies to combine distilled data with raw parallel data, helping models "remember" rare words without losing the speed benefits of NAT.
- Selective Distillation: Min Liu and Shujian Huang proposed using evaluators to identify high-quality "raw" samples that are easy for the student to learn, bypassing the need for a teacher's simplified output for every sentence.
The Efficiency Trade-off
While KD is essential for model compression and NAT, it is not a "free lunch." Goran Glavaš and Fabian David Schmidt point out that vanilla distillation can underperform compared to training from scratch on tasks requiring high syntactic sensitivity. The challenge for future research, as summarized in the survey by Ona de Gibert and Jörg Tiedemann, is to balance the "supervision shaping" benefits of KD with the need to preserve the rich, complex nuances of human language.