Application of Knowledge Distillation Based on Transfer Learning of ERNIE Model in Intelligent Dialogue Intention Recognition

Shiguang Guo, Qing WangSensors 2022 · DOI

TL;DR

This paper tackles the challenge of making advanced AI chatbots more efficient for real-time use. They propose using knowledge distillation to transfer the understanding capabilities of a large, powerful ERNIE model into a much faster FastText model, significantly improving both accuracy and speed for dialogue intention recognition.

What Problem Does This Paper Address?

Task-oriented chatbots, crucial for services like e-commerce customer support, rely heavily on accurately understanding user intentions. While powerful pre-trained language models like BERT and ERNIE excel at text classification tasks, their large size and computational demands make them impractical for high-throughput, real-time applications (high QPS). Conversely, simpler models like FastText are fast but often lack the accuracy of their larger counterparts. This creates a trade-off between performance and accuracy, hindering the deployment of sophisticated chatbots in demanding environments.

Methods

The core of this paper is knowledge distillation, a technique where a smaller "student" model learns from a larger, more capable "teacher" model. Here, the ERNIE model acts as the teacher, and FastText is the student.

  1. Teacher Model (ERNIE): ERNIE (Enhanced Representation through Knowledge Integration) is a pre-trained language model that builds upon BERT by incorporating enhanced masking strategies (phrase-based and entity-based) to better capture semantic relationships. The authors use a pre-trained ERNIE 1.0 base model.
  2. Student Model (FastText): FastText is a well-known model for text classification that offers good computational efficiency. It represents text using word embeddings and a "bag of words" approach, making it fast for inference.
  3. Knowledge Distillation Process:
    • The ERNIE model is used to predict intention labels for a large amount of unlabeled online data. This process generates "soft labels" (probability distributions over classes) rather than just hard labels.
  4. These soft labels from ERNIE, which encapsulate richer information than one-hot encoded labels, are then used to train the FastText student model.
  5. A "heating technique" is applied to the softmax function in the distillation process, which involves a temperature parameter (T) to smooth the probability distribution, allowing the student model to learn more effectively from the teacher's nuanced predictions.

The paper also compares their approach against other models like LSTM and BERT.

Datasets

  • Training Corpus: A manually labeled corpus of 120,000 customer service Q&A entries for e-commerce after-sales scenarios. This corpus covers 16 task-oriented intention recognition categories, including common intents like "Urging Delivery," "Request a Replacement," and "Return Shipping Fee," along with a category for non-business (chitchat) intentions.
  • Distillation Data: One million online real customer service chat corpora were used for distillation. This data was unlabeled and processed by the ERNIE teacher model to generate soft labels.
  • Test Data: Approximately 10,000 samples were used as a test set.

Key Insights and Ideas

  • Bridging the Performance-Accuracy Gap: The primary insight is that knowledge distillation can effectively bridge the gap between computationally expensive, high-accuracy models (like ERNIE) and fast, efficient models (like FastText) for real-time chatbot applications.
  • Soft Labels are Powerful: Using the probability distributions (soft labels) from the teacher model provides richer supervisory signals than traditional hard labels, leading to better learning for the student model.
  • ERNIE's Advantage: ERNIE's enhanced masking strategies seem to provide a more robust understanding of semantic dependencies, making it a more effective teacher model compared to BERT for this specific task.
  • Computational Efficiency is Key for Chatbots: The paper highlights that for high-concurrency scenarios, computational performance is as critical as accuracy, making distillation a practical solution.

Key Results

  • The ERNIE base model achieved the highest accuracy among the tested models (95.7%) on the e-commerce intention recognition task.
  • FastText, when distilled by ERNIE, achieved an accuracy of 95.1%, a significant improvement over its native performance (94.16%) and close to the teacher model's accuracy.
  • Crucially, the distilled FastText model maintained excellent computational performance, with a time-consuming of 0.1 ms, making it suitable for high QPS environments, unlike ERNIE or BERT which had significantly higher time costs (10.0+ ms).
  • The paper also explored the impact of FastText's internal parameters (dimension, n-grams) on accuracy, showing improvements with larger dimensions and specific n-gram configurations.

Limitations and Future Work

  • Model Complexity of Teacher: The paper acknowledges that while ERNIE provides superior accuracy, its complex network structure and high computational requirements (especially GPU costs) make it difficult to meet online performance demands directly. This is precisely why distillation is necessary.
  • Data Requirements: The distillation process relies on a large amount of unlabeled data for the teacher model to process.
  • Future Work: The paper implicitly suggests that further exploration of distillation techniques, potentially with more advanced ERNIE versions or other teacher models, could yield even better results. Optimizing the distillation process itself (e.g., temperature tuning, different distillation objectives) could also be a direction. The focus on e-commerce after-sales service also leaves room for applying this approach to other domains.

Summary updated

Abstract

The 'intention' classification of a user question is an important element of a task-engine driven chatbot. The essence of a user question's intention understanding is the text classification. The transfer learning, such as BERT (Bidirectional Encoder Representations from Transformers) and ERNIE (Enhanced Representation through Knowledge Integration), has put the text classification task into a new level, but the BERT and ERNIE model are difficult to support high QPS (queries per second) intelligent dialogue systems due to computational performance issues. In reality, the simple classification model usually shows a high computational performance, but they are limited by low accuracy. In this paper, we use knowledge of the ERNIE model to distill the FastText model; the ERNIE model works as a teacher model to predict the massive online unlabeled data for data enhancement, and then guides the training of the student model of FastText with better computational efficiency. The FastText model is distilled by the ERNIE model in chatbot intention classification. This not only guarantees the superiority of its original computational performance, but also the intention classification accuracy has been significantly improved.
BibTeX
@inproceedings{guo2022application,
  author = {Shiguang Guo and Qing Wang},
  title = {{Application of Knowledge Distillation Based on Transfer Learning of ERNIE Model in Intelligent Dialogue Intention Recognition}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22031270},
}
Figure