Meta-learning for Classifying Previously Unseen Data Source into Previously Unseen Emotional Categories

Gaël Guibon, Matthieu Labeau, Hélène Flamein, Luce Lefeuvre, Chloé ClavelACL 2021

TL;DR

This paper tackles the challenge of emotion classification in private, unlabeled messages by leveraging meta-learning. By training on publicly available social media data, the authors demonstrate that a meta-learned model can effectively classify emotions in a new, unseen dataset with different emotional categories and data characteristics, even outperforming traditional supervised methods in this cross-domain scenario.

What Problem Does This Paper Address?

Training accurate emotion classifiers for specialized, private data (like company messages) is difficult due to the high cost and complexity of annotation, as well as privacy concerns. This means we often have limited labeled data for our specific target domain. Furthermore, the target domain might use a different set of emotion labels or have distinct linguistic features compared to readily available public datasets. This paper addresses the problem of building an emotion classifier for such a scenario, where the target data's labels and even its source are unknown during the initial training phase.

Methods

The core of this work is meta-learning, specifically a few-shot learning approach using Prototypical Networks. The authors frame emotion classification as a series of "tasks," where each task involves classifying a small number of examples.

  • Meta-Training: The model is trained on a large dataset (GoEmotions) using an episodic approach. Each episode simulates a few-shot learning scenario: a small "support set" of labeled examples is used to learn a class prototype (an average representation of the class), and then a "query set" of unlabeled examples is used to classify based on their distance to these prototypes.
  • Encoder: A Transformer encoder is used to generate rich embeddings for the text, capturing nuanced meaning. This encoder is the component that is meta-learned.
  • Metric Learning: Prototypical Networks are employed, which learn a distance metric (Euclidean distance in this case) to compare query examples to class prototypes.
  • Two-Level Meta-Learning: The approach involves learning from a source dataset (GoEmotions) with a specific set of labels and then transferring this knowledge to a target dataset (DailyDialog) with a different set of labels and linguistic characteristics.

Datasets

  • GoEmotions: A large dataset of 58,000 Reddit comments labeled with 27 emotion categories. This dataset was used for meta-training and hyper-parameter tuning. It was split into three subsets (EmoTagSets) for different stages of the meta-learning process.
  • DailyDialog: A dataset of 13,118 human-written daily conversations. For this study, individual utterances were used, labeled with 6 emotion categories. This served as the target dataset for evaluating the meta-learned model.

Key Insights and Ideas

  • Meta-learning bridges the domain gap: The study demonstrates that meta-learning can effectively transfer knowledge across different data sources and label sets, a significant challenge in NLP.
  • Linguistic structure matters more than label overlap: The results suggest that differences in linguistic structure and vocabulary between the source and target datasets pose a greater challenge than simply having different label sets.
  • Transformer encoders are crucial for meta-learning: The Transformer encoder, with its attention mechanism, proved to be the most effective for capturing the necessary meta-information for cross-domain emotion classification.
  • Fine-tuning can hurt meta-learning: Surprisingly, fine-tuning the meta-learned model on the target dataset's labels led to worse performance, indicating that the meta-learned representations are sensitive to such adjustments.

Key Results

  • The meta-learning approach achieved competitive results on the DailyDialog dataset, even outperforming a naive supervised baseline when applied to this unseen data source and label set.
  • Specifically, the Transformer encoder with Prototypical Networks achieved an F1-score of 58.55% on the DailyDialog test set, while a naive supervised model trained on GoEmotions and tested on DailyDialog only achieved 34.58%.
  • When comparing different encoders, the Transformer encoder consistently outperformed average word embeddings (AVG) and Convolutional Neural Networks (CNN) in the meta-learning setting.
  • The study also explored fine-tuning pre-trained language models like BERT, XLNet, and RoBERTa, finding BERT to be the most promising, but still less effective than static embeddings when meta-learned.

Limitations and Future Work

  • Data Source Differences: The authors acknowledge that differences in linguistic structure and conversational context between social media (GoEmotions) and dyadic conversations (DailyDialog) are a primary source of error and limit performance.
  • Conversational Context: The current approach treats each utterance independently, ignoring the conversational context, which could provide valuable information for emotion classification.
  • Future Work: The authors propose investigating the integration of conversational context into the meta-learning framework and extending the approach to other languages, such as English. They also suggest exploring other meta-learning strategies and encoders.

Summary updated

BibTeX
@inproceedings{guibon2021meta,
  author = {Gaël Guibon and Matthieu Labeau and Hélène Flamein and Luce Lefeuvre and Chloé Clavel},
  title = {{Meta-learning for Classifying Previously Unseen Data Source into Previously Unseen Emotional Categories}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure