Abstract
Traditional Knowledge Distillation (KD) requires a pre-trained teacher model to provide soft targets for a student model. This paper introduces Self-Knowledge Distillation (SKD), a novel approach where an NLP model acts as its own teacher. By leveraging the semantic relationships within its own word embedding space, the model generates soft targets dynamically during training. Experimental results on language modeling and neural machine translation tasks demonstrate that SKD improves performance, enhances semantic coherence, and provides effective regularization against overfitting.
Introduction
In Natural Language Processing (NLP), models are typically trained using cross-entropy loss against one-hot labels. This objective treats all incorrect predictions equally, ignoring the inherent semantic similarities between words. While traditional KD addresses this by using a large teacher model to provide a richer probability distribution, it introduces significant computational overhead. Self-Knowledge Distillation (SKD) aims to capture these semantic nuances "on the fly" without a separate teacher model.
Methodology: Self-Knowledge Distillation (SKD)
The core of SKD lies in utilizing the model's internal word embedding space to define similarity. Words with similar meanings naturally cluster together in this space.
Soft Target Generation
Instead of a rigid one-hot vector, SKD calculates a "soft" probability distribution. The similarity between a target word ($w_t$) and a predicted word ($w_n$) is estimated using Euclidean distance in the embedding space. To maintain computational efficiency, the authors employ a two-class approximation, focusing only on the ground-truth target and the model's current top prediction.
Dynamic Scheduling and Constraints
To ensure stability, the probability of the predicted word is clipped at 0.5, ensuring it never exceeds the target word's importance. Furthermore, a dynamic scheduling parameter, $\alpha$, is introduced. This parameter gradually shifts the training objective from standard cross-entropy to SKD as the model's embeddings become more reliable over time.
Experimental Results
The researchers evaluated SKD across Language Modeling (LM) and Neural Machine Translation (NMT) tasks using datasets such as Penn TreeBank (PTB), Wiki-2, and WMT’15.
Performance Gains
In language modeling on the PTB dataset, the Negative Log Likelihood (NLL) improved from 101.40 (baseline) to 99.38 with SKD. In NMT tasks, SKD consistently yielded higher BLEU scores; for instance, English-to-Finnish translation scores rose from 9.01 to 9.87.
Regularization and Semantic Coherence
Beyond quantitative metrics, SKD acts as a powerful regularizer. By providing a smoother target distribution, it prevents overconfidence and delays the onset of overfitting. Qualitative analysis of translation outputs revealed that SKD-trained models produce more semantically sensible sentences, even when errors occur, by favoring "near-miss" synonyms over unrelated words.
Conclusion
Self-Knowledge Distillation offers a computationally efficient way to incorporate semantic knowledge into NLP training. By allowing a model to learn from its own internal representations, SKD achieves superior accuracy and robustness. Future research may explore generalizing this approach to multi-class soft targets and applying similar principles to other domains like computer vision.
Summary updated