Self-Knowledge Distillation in Natural Language Processing

Sangchul Hahn, Heeyoul ChoiarXiv 2019 · arXiv

Abstract

Traditional Knowledge Distillation (KD) requires a pre-trained teacher model to provide soft targets for a student model. This paper introduces Self-Knowledge Distillation (SKD), a novel approach where an NLP model acts as its own teacher. By leveraging the semantic relationships within its own word embedding space, the model generates soft targets dynamically during training. Experimental results on language modeling and neural machine translation tasks demonstrate that SKD improves performance, enhances semantic coherence, and provides effective regularization against overfitting.

Introduction

In Natural Language Processing (NLP), models are typically trained using cross-entropy loss against one-hot labels. This objective treats all incorrect predictions equally, ignoring the inherent semantic similarities between words. While traditional KD addresses this by using a large teacher model to provide a richer probability distribution, it introduces significant computational overhead. Self-Knowledge Distillation (SKD) aims to capture these semantic nuances "on the fly" without a separate teacher model.

Methodology: Self-Knowledge Distillation (SKD)

The core of SKD lies in utilizing the model's internal word embedding space to define similarity. Words with similar meanings naturally cluster together in this space.

Conceptual diagram of Self-Knowledge Distillation (SKD) showing the transition from one-hot targets to soft targets based on embedding space proximity.

Soft Target Generation

Instead of a rigid one-hot vector, SKD calculates a "soft" probability distribution. The similarity between a target word ($w_t$) and a predicted word ($w_n$) is estimated using Euclidean distance in the embedding space. To maintain computational efficiency, the authors employ a two-class approximation, focusing only on the ground-truth target and the model's current top prediction.

Dynamic Scheduling and Constraints

To ensure stability, the probability of the predicted word is clipped at 0.5, ensuring it never exceeds the target word's importance. Furthermore, a dynamic scheduling parameter, $\alpha$, is introduced. This parameter gradually shifts the training objective from standard cross-entropy to SKD as the model's embeddings become more reliable over time.

Experimental Results

The researchers evaluated SKD across Language Modeling (LM) and Neural Machine Translation (NMT) tasks using datasets such as Penn TreeBank (PTB), Wiki-2, and WMT’15.

Training curves comparing the baseline model with Noise and SKD variants, showing improved performance and stability.

Performance Gains

In language modeling on the PTB dataset, the Negative Log Likelihood (NLL) improved from 101.40 (baseline) to 99.38 with SKD. In NMT tasks, SKD consistently yielded higher BLEU scores; for instance, English-to-Finnish translation scores rose from 9.01 to 9.87.

Regularization and Semantic Coherence

Beyond quantitative metrics, SKD acts as a powerful regularizer. By providing a smoother target distribution, it prevents overconfidence and delays the onset of overfitting. Qualitative analysis of translation outputs revealed that SKD-trained models produce more semantically sensible sentences, even when errors occur, by favoring "near-miss" synonyms over unrelated words.

Conclusion

Self-Knowledge Distillation offers a computationally efficient way to incorporate semantic knowledge into NLP training. By allowing a model to learn from its own internal representations, SKD achieves superior accuracy and robustness. Future research may explore generalizing this approach to multi-class soft targets and applying similar principles to other domains like computer vision.

Summary updated

Abstract

Since deep learning became a key player in natural language processing (NLP), many deep learning models have been showing remarkable performances in a variety of NLP tasks, and in some cases, they are even outperforming humans. Such high performance can be explained by efficient knowledge representation of deep learning models. While many methods have been proposed to learn more efficient representation, knowledge distillation from pretrained deep networks suggest that we can use more information from the soft target probability to train other neural networks. In this paper, we propose a new knowledge distillation method self-knowledge distillation, based on the soft target probabilities of the training model itself, where multimode information is distilled from the word embedding space right below the softmax layer. Due to the time complexity, our method approximates the soft target probabilities. In experiments, we applied the proposed method to two different and fundamental NLP tasks: language model and neural machine translation. The experiment results show that our proposed method improves performance on the tasks.
BibTeX
@misc{hahn2019self,
  author = {Sangchul Hahn and Heeyoul Choi},
  title = {{Self-Knowledge Distillation in Natural Language Processing}},
  year = {2019},
  eprint = {1908.01851},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.01851},
}
Figure