Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization

Yangyang Shi, Mei-Yuh Hwang, Xin Lei, Haoyu Sheng

Abstract

Recurrent Neural Networks (RNNs) have dominated language modeling because of their superior performance over traditional N-gram based models. In many applications, a large Recurrent Neural Network language model (RNNLM) or an ensemble of several RNNLMs is used. These models have large memory footprints and require heavy computation. In this paper, we examine the effect of applying knowledge distillation in reducing the model size for RNNLMs. In addition, we propose a trust regularization method to improve the knowledge distillation training for RNNLMs. Using knowledge distillation with trust regularization, we reduce the parameter size to a third of that of the previously published best model while maintaining the state-of-the-art perplexity result on Penn Treebank data. In a speech recognition N-best rescoring task, we reduce the RNNLM model size to 18.5% of the baseline system, with no degradation in word error rate (WER) performance on Wall Street Journal data set.

Introduction

Language modeling is a fundamental component in various natural language processing tasks, including speech recognition and machine translation. While Recurrent Neural Networks (RNNs) significantly outperform traditional N-gram models, the highest-performing RNNLMs are often large ensembles or high-rank models. These "heavyweight" architectures present significant challenges for deployment on resource-constrained devices due to their substantial memory footprints and computational requirements.

Knowledge Distillation (KD) is a popular technique for model compression, where a smaller "student" model is trained to mimic the behavior of a larger "teacher" model. However, standard KD methods—typically effective in computer vision—often underperform when applied to language modeling. Specifically, the common practice of using a fixed interpolation between "hard" training labels and "soft" teacher predictions can lead to suboptimal results.

Proposed Method: Trust Regularization

To address the limitations of standard distillation in the context of RNNLMs, this work introduces Trust Regularization (TR).

The Distillation Framework

The approach utilizes a high-rank RNNLM teacher, specifically one employing a "Mixture of Softmaxes" (MoS) to capture complex linguistic nuances. The student is a significantly more compact LSTM model.

Trust Regularization Mechanism

Unlike traditional KD, which uses a static weight to balance cross-entropy (CE) loss from data and Kullback-Leibler (KL) divergence from the teacher, TR dynamically adjusts this balance. The core logic is based on the teacher's confidence:

  • High Trust: If the teacher's prediction aligns strongly with the ground-truth "hard" label, the system increases the weight of the CE loss, trusting the data more.
  • Low Trust: If the teacher is uncertain or its prediction diverges from the label, the student relies more heavily on the teacher's full probability distribution (KL loss).

Optimization Strategies

Empirical findings revealed that standard regularization techniques essential for training the teacher, such as Dropout and Activation Regularization, should be disabled for the student during distillation. Removing these stochastic elements allows the student to focus more effectively on capturing the "dark knowledge" provided by the teacher.

Experimental Setup

The proposed method was evaluated on two benchmark datasets:

  1. Penn Treebank (PTB): A classic dataset for measuring language model perplexity.
  2. Wall Street Journal (WSJ): Used for an N-best rescoring task in speech recognition to evaluate Word Error Rate (WER).

Key Results

The experimental results demonstrate significant improvements in model efficiency without performance loss.

Penn Treebank Performance

On the PTB dataset, the student model achieved state-of-the-art perplexity with only 7 million parameters. This represents a reduction to nearly one-third the size of the previous best-published model (22 million parameters).

Comparison of RNNLM models on the Penn Treebank (PTB) dataset.

Speech Recognition Efficiency

In the WSJ rescoring task, the student model was compressed to 18.5% of the teacher's parameter count (12M vs. 65M). Despite this drastic reduction, the student maintained a WER nearly identical to the teacher, significantly outperforming models trained on data alone.

Performance of distilled models on the Wall Street Journal (WSJ) speech recognition task.

Dynamic Evaluation

When combined with dynamic evaluation—updating model weights during the test phase—the 7M parameter student model achieved a test perplexity of 47.60, rivaling much larger and more complex architectures.

Conclusion

This work demonstrates that Trust Regularization effectively bridges the gap between large-scale teacher models and efficient student models in language modeling. By dynamically weighting the teacher's guidance and simplifying the student's training objective, researchers can achieve massive compression ratios. While this study focuses on RNNs, the principles of dynamic trust in distillation provide a promising direction for future research in compressing modern architectures like Transformers.

Summary updated

Abstract

Recurrent Neural Networks (RNNs) have dominated language modeling because of their superior performance over traditional N-gram based models. In many applications, a large Recurrent Neural Network language model (RNNLM) or an ensemble of several RNNLMs is used. These models have large memory footprints and require heavy computation. In this paper, we examine the effect of applying knowledge distillation in reducing the model size for RNNLMs. In addition, we propose a trust regularization method to improve the knowledge distillation training for RNNLMs. Using knowledge distillation with trust regularization, we reduce the parameter size to a third of that of the previously published best model while maintaining the state-of-the-art perplexity result on Penn Treebank data. In a speech recognition N-bestrescoring task, we reduce the RNNLM model size to 18.5% of the baseline system, with no degradation in word error rate(WER) performance on Wall Street Journal data set.
BibTeX
@inproceedings{shi2019knowledge,
  author = {Yangyang Shi and Mei-Yuh Hwang and Xin Lei and Haoyu Sheng},
  title = {{Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8683533},
}
Figure