Knowledge Distillation from Monolingual to Multilingual Models for Intelligent and Interpretable Multilingual Emotion Detection

Yuqi Wang, Zimu Wang, Nijia Han, Wei Wang, Qi Chen, Haiyang Zhang, Yushan Pan, Anh NguyenWASSA 2024 · OpenReview

TL;DR

This paper tackles the challenge of multilingual emotion detection, which suffers from data scarcity in many languages and a lack of interpretability. They propose a novel approach using knowledge distillation from a strong English monolingual model to a multilingual model, significantly improving performance and enabling better identification of emotion-trigger words.

What Problem Does This Paper Address?

Emotion detection from text is vital for understanding human sentiment across various applications, from social media analysis to customer feedback. However, building effective multilingual emotion detection systems is hindered by two major issues:

  1. Data Scarcity: Most existing research focuses on high-resource languages like English, where ample labeled data is available. Many other languages lack sufficient training data, making it difficult to build robust models for them.
  2. Lack of Interpretability: Current models often excel at predicting emotions but struggle to explain why they made a particular prediction. Understanding the specific words or phrases that trigger an emotion is crucial for trust and debugging.

This paper aims to bridge this gap by developing a method that enhances both the performance and interpretability of multilingual emotion detection, particularly in low-resource scenarios.

Methods

The core of the proposed method lies in a two-stage process leveraging knowledge distillation:

  1. Parallel Model Adaptation:
    • A high-performing English monolingual model (e.g., RoBERTa) is fine-tuned on an English emotion detection dataset. This model, denoted as $\theta_{en}$, learns English-specific linguistic nuances and emotion cues.
  2. Concurrently, a multilingual pre-trained language model (PLM) like XLM-RoBERTa ($\theta_{mul}$) is also fine-tuned on the same English dataset. This allows the multilingual model to adapt its cross-lingual representations to the emotion detection task while retaining its ability to generalize across languages.
  3. Knowledge Distillation:
    • To transfer the specialized knowledge from the English monolingual "teacher" model to the multilingual "student" model, a knowledge distillation strategy is employed.
  4. Non-English development set instances are translated into English using neural machine translation.
  5. The monolingual model predicts emotions on the translated English data, while the multilingual model predicts on the original (potentially non-English) data.
  6. A "consistency loss" is computed using Kullback-Leibler (KL) divergence between the output distributions of the two models. This loss encourages the multilingual model to mimic the teacher's predictions.
  7. Crucially, a Translation Quality Weighting mechanism is introduced. This weighting scheme uses the disagreement (measured by Mean Squared Error) between the multilingual model's predictions on original and translated data to down-weight instances where translation quality is suspected to be poor. This ensures that knowledge transfer primarily occurs from reliable translations.

Datasets

The primary dataset used for evaluation is the EXALT (Explainable Cross-lingual Emotion Detection in Tweets) dataset.

  • Training Set: English only.
  • Development and Test Sets: Include five languages: English, French, Dutch, Russian, and Spanish.
  • Key Feature: This dataset provides annotations for both overall expressed emotion and word triggers, making it suitable for evaluating both emotion detection and explainability.

Key Insights and Ideas

  • Monolingual Strength for Multilingual Gain: The paper demonstrates that a strong monolingual model, even if specialized for one language, can serve as an invaluable "teacher" to significantly boost the performance of a more general multilingual model.
  • Targeted Knowledge Transfer: Knowledge distillation isn't just about copying predictions; the translation quality weighting mechanism is a clever way to ensure that the student model learns from the most reliable "lessons" from the teacher, avoiding noise from poor translations.
  • Interpretability Boost: The distillation process not only improves emotion detection accuracy but also enhances the models' ability to identify emotion-trigger words, a critical aspect for building trustworthy AI systems.
  • Generative vs. Discriminative Models: The results suggest that for tasks requiring nuanced understanding like emotion detection, large discriminative models (like XLM-RoBERTa) can outperform even larger generative models (like BLOOM), highlighting the importance of model architecture and training strategy over sheer parameter count.

Key Results

  • Performance Gains: After distillation, multilingual models like XLM-RoBERTa and E5 showed significant improvements, with an average F1 score increase of 4.87% on the development set.
  • Explainability Improvement: The refined XLM-RoBERTa and E5 models exhibited better zero-shot performance on binary trigger detection tasks compared to their original versions, indicating improved ability to pinpoint emotion-trigger words.
  • Varied Impact: While XLM-ROBERTa and E5 benefited substantially, the BGE-M3 model showed only a minor improvement, suggesting that its multi-functional pre-training might lead to representations less aligned for specific emotion detection tasks.

Limitations and Future Work

Limitations:

  • Translation System Dependency: The effectiveness of the knowledge distillation process is still reliant on the quality of the machine translation system used for translating data.
  • Computational Cost: Training multiple models and performing additional inference steps for weighting and distillation can be computationally intensive.

Future Work:

  • Advanced Distillation Techniques: Exploring more sophisticated knowledge distillation methods.
  • Improved Translation Methods: Investigating more accurate and effective translation techniques to further enhance the distillation process.

Summary updated

Abstract

Emotion detection from text is a crucial task in understanding natural language with wide-ranging applications. Existing approaches for multilingual emotion detection from text face challenges with data scarcity across many languages and a lack of interpretability. We propose a novel method that leverages both monolingual and multilingual pre-trained language models to improve performance and interpretability. Our approach involves 1) training a high-performing English monolingual model in parallel with a multilingual model and 2) using knowledge distillation to transfer the emotion detection capabilities from the monolingual teacher to the multilingual student model. Experiments on a multilingual dataset demonstrate significant performance gains for refined multilingual models like XLM-RoBERTa and E5 after distillation. Furthermore, our approach enhances interpretability by enabling better identification of emotion-trigger words. Our work presents a promising direction for building accurate, robust and explainable multilingual emotion detection systems.
BibTeX
@inproceedings{wang2024knowledge,
  author = {Yuqi Wang and Zimu Wang and Nijia Han and Wei Wang and Qi Chen and Haiyang Zhang and Yushan Pan and Anh Nguyen},
  title = {{Knowledge Distillation from Monolingual to Multilingual Models for Intelligent and Interpretable Multilingual Emotion Detection}},
  booktitle = {WASSA},
  year = {2024},
  url = {https://openreview.net/forum?id=KOzgdHJ7Zx},
}
Figure