Distilling Multilingual Transformers into CNNs for Scalable Intent Classification

Besnik Fetahu, Akash Veeragouni, Oleg Rokhlenko, Shervin MalmasiEMNLP 2022 · DOI

TL;DR

This paper tackles the challenge of deploying large, state-of-the-art multilingual Transformer models for real-time voice assistant tasks like intent classification. They achieve this by distilling knowledge from these powerful but slow Transformers into much smaller, faster Convolutional Neural Networks (CNNs) without sacrificing accuracy, enabling significant speedups and cost reductions.

What Problem Does This Paper Address?

Modern voice assistants rely heavily on understanding user intent (e.g., "What's the weather like?" or "Play my favorite song"). Transformer models, especially multilingual versions like XLM-R, have revolutionized Natural Language Understanding (NLU) tasks, offering impressive accuracy across many languages. However, their massive size (hundreds of gigabytes) and slow inference times make them impractical for real-time applications that need to process millions of requests per minute with low latency. This paper addresses the critical gap between the performance of large Transformer models and the deployment constraints of real-world, low-latency systems, particularly for multilingual intent classification.

Methods

The core of this paper is Knowledge Distillation (KD), a technique where a smaller "student" model learns to mimic the behavior of a larger, pre-trained "teacher" model. Specifically, they employ a cross-architecture distillation approach, transferring knowledge from a Transformer teacher to a CNN student.

Key aspects of their method include:

  • Teacher Model: A multilingual XLM-RoBERTa (XLM-R) base model, fine-tuned for the intent classification task.
  • Student Model: A deep Convolutional Neural Network (CNN) architecture. CNNs are chosen for their efficiency in text classification and ability to capture local subword interactions.
  • Distillation Objective: Instead of using the teacher's final probabilities, they use the unscaled logits (raw outputs before softmax) as "soft targets." This logit-based distillation is more numerically stable and effective for transferring nuanced knowledge. The objective is to minimize the Mean Squared Error (MSE) between the teacher's and student's logits.
  • Unsupervised Data: Crucially, the distillation process uses unlabelled data. This allows the student to learn not only from the explicit intent labels but also from the broader linguistic knowledge embedded in the Transformer's pre-trained embeddings and self-attention mechanisms, which might not be covered by the supervised training data alone.

Datasets

The paper utilizes several types of data:

  • Teacher Datasets: Supervised data used to train the teacher model. This dataset contains 3.5 million utterances across 7 languages and 11 locales.
  • Distillation Datasets: Unlabelled utterances used for the knowledge distillation process. This dataset is significantly larger, comprising 22.6 million utterances sampled over a month. The authors investigate the impact of varying the amount of this unlabelled data.
  • Test Datasets: Labelled data used to evaluate the performance of both the teacher and student models. This set consists of 1.7 million labelled instances across all locales.

The paper does not introduce new benchmark datasets but rather uses existing data for a real-world application.

Key Insights and Ideas

  • Logit Distillation is Key: Using unscaled logits from the teacher model is more effective for transferring knowledge than using final probabilities, especially for cross-architecture distillation. This allows the student to learn richer representations.
  • Unsupervised Data is Powerful: Distilling using unlabelled data allows the student CNN to capture a broader range of knowledge from the Transformer teacher, leading to better generalization than training a CNN directly on labelled data.
  • CNNs are Viable for Multilingual IC: Despite the dominance of Transformers, well-designed CNNs, when distilled, can achieve comparable accuracy to large multilingual Transformers for intent classification, while being vastly more efficient.
  • Efficiency Gains are Massive: The proposed distilled CNN model achieves an 18x speedup in inference latency compared to the teacher Transformer, making real-time, global deployment feasible.

Key Results

  • Accuracy: The distilled student model (S) achieves performance nearly identical to the teacher model (T), with an average PR-BEP gap of only 0.1%. This is in stark contrast to a student model trained directly on supervised data (Ssup), which showed a significant performance gap (average 6%).
  • Inference Latency: The student model achieves an inference latency of just 2.7ms, an 18x improvement over the teacher Transformer. This is also significantly faster than baseline QRNN models (3.8x and 2.95x faster than QRNN5 and QRNN4, respectively).
  • Data Efficiency: Even with a small fraction (1%) of the unlabelled distillation data (approx. 2.2M instances), the student model achieves comparable performance to the teacher, with a PR-BEP gap of less than 1% when using 10% of the data.
  • Generalization: The student model demonstrates similar generalization power to the teacher on unseen data, with low KL divergence in output probabilities.

Limitations and Future Work

  • Limitations:
    • The paper focuses on intent classification; its applicability to other NLU tasks might require further investigation.
  • While the CNN architecture is efficient, it might not capture the same depth of long-range dependencies as Transformers, though distillation mitigates this.
  • Future Work:
    • The authors suggest exploring other distillation strategies and student architectures.
  • Investigating the impact of distillation on other NLU tasks beyond intent classification.
  • Further exploring the optimal amount of unlabelled data needed for effective distillation across different languages and tasks.

Summary updated

BibTeX
@inproceedings{fetahu2022distilling,
  author = {Besnik Fetahu and Akash Veeragouni and Oleg Rokhlenko and Shervin Malmasi},
  title = {{Distilling Multilingual Transformers into CNNs for Scalable Intent Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-industry.43},
}
Figure