Pseudo-Labeling With Large Language Models for Multi-Label Emotion Classification of French Tweets

Usman Malik, Simon Bernard, Alexandre Pauchet, Clément Chatelain, Romain Picot-Clémente, Jérôme CortinovisIEEE Access 2024 · OpenReview

Abstract

This study proposes a novel semi-supervised multi-label emotion classification approach for French tweets based on pseudo-labeling. Human subjectivity in emotional expression makes it difficult for a machine to learn, necessitating large datasets annotated by multiple individuals. However, creating such datasets is costly and time-consuming. This work introduces a novel approach by leveraging pseudo-labels generated through ChatGPT. Using zero-shot, one-shot, and few-shot learning techniques, ChatGPT annotates the unlabeled portion of a dataset, which is then merged with manual annotations to train a classification model. We present a new French tweet dataset, EmoDIFT, containing 2,350 tweets related to an urban industrial incident. Our findings demonstrate that this approach outperforms standard pseudo-labeling and established baselines, particularly for nuanced emotional states.

Introduction

Emotion classification is a complex task that extends beyond simple sentiment analysis. While sentiment analysis typically categorizes text as positive, negative, or neutral, emotion classification must account for a spectrum of overlapping feelings such as anger, mistrust, and sadness. This complexity is compounded in short-form text like tweets, where context is limited and subjectivity is high.

A significant barrier to developing robust emotion classifiers is the lack of large-scale, high-quality annotated datasets, especially for languages other than English. Manual annotation is expensive and often suffers from low inter-annotator agreement. To address this, we propose a semi-supervised learning framework that utilizes Large Language Models (LLMs) to generate high-quality "pseudo-labels" for unlabeled data, effectively augmenting small "gold standard" datasets.

The EmoDIFT Dataset

Central to this research is the introduction of EmoDIFT, a specialized dataset focused on the 2019 Lubrizol factory fire in Rouen, France. The dataset consists of:

  • 2,350 manually annotated tweets: Each tweet was evaluated by three human annotators across eight emotional categories: Anger, Disgust, Fear, Mistrust, Sadness, Surprise, Irony, and Neutral.
  • Aggregation Method: To handle subjectivity, annotations were aggregated using a method that transforms ranks into "degrees of belonging," capturing the collective emotional state rather than a simple majority vote.
  • 2,800 unlabeled tweets: These were reserved for the pseudo-labeling process to expand the training set.

Methodology: LLM-Augmented Pseudo-Labeling

The core of our approach is a semi-supervised pipeline that leverages the linguistic capabilities of ChatGPT (GPT-3.5 Turbo) to act as an expert annotator.

1. Prompting Strategies

We evaluated three distinct prompting techniques to generate pseudo-labels:

  • Zero-shot: The LLM is provided with the list of emotions and asked to classify the tweet without examples.
  • One-shot: The LLM is provided with a single labeled example to establish context.
  • Few-shot: The LLM is provided with two examples per emotion.

2. Model Training

The LLM-generated pseudo-labels were combined with the manual "gold" labels. This augmented dataset was used to fine-tune several transformer-based models, including CamemBERT, FlauBERT, and RoBERTa, integrated with a Deep Neural Network (DNN) for multi-label classification. This process effectively distills the broad knowledge of the LLM into a specialized, efficient model for French crisis communication.

Results and Analysis

Our experiments compared the LLM-augmented models against fully supervised baselines and traditional pseudo-labeling techniques (where a standard classifier generates the labels).

  • Performance Gains: The CamemBERT-Large model trained with few-shot pseudo-labels achieved the highest performance with an F1 score of 0.6662, surpassing the fully supervised baseline (0.6538) and standard pseudo-labeling (0.6310).
  • Prompting Efficiency: While few-shot prompting yielded the best results, one-shot prompting provided a significant improvement over zero-shot with only a marginal performance gap compared to few-shot. This suggests that one-shot prompting is the most cost-effective strategy for large-scale applications.
  • Nuanced Emotions: The LLM-assisted approach was particularly effective at identifying rare or underrepresented emotions that traditional supervised models often miss due to data sparsity.
  • The "Surprise" Exception: Interestingly, performance on the "Surprise" emotion decreased slightly with pseudo-labeling. This may be due to the rarity of this emotion in the LLM's training data or a misalignment between the LLM's interpretation of surprise and that of the human annotators.

Conclusion

This research demonstrates that LLMs can serve as powerful proxies for expert annotators in specialized domains. By using ChatGPT to generate pseudo-labels, we can significantly improve the performance of emotion classification models while reducing the reliance on costly manual labor. Future work will explore more advanced models like GPT-4 and sophisticated "chain-of-thought" prompting to further refine the capture of emotional nuances in social media data.

Summary updated

Abstract

This study proposes a novel semi-supervised multi-label emotion classification approach for French tweets based on pseudo-labeling. Human subjectivity in emotional expression makes it difficult for a machine to learn. Therefore, it necessitates training supervised learning models on large datasets annotated by multiple annotators. However, creating such datasets can be costly and time-consuming. Moreover, aggregating annotations from multiple annotators to capture their collective emotional state adds complexity to the task. Semi-supervised learning techniques have shown effectiveness with limited datasets. Furthermore, Large language Models (LLMs), particularly Chat-GPT, have demonstrated superior annotation accuracy compared to annotations obtained from crowdsourcing platforms, when both are evaluated against expert-annotated data. This work introduces a novel approach for multi-label emotion classification of French tweets by leveraging pseudo-labels generated through Chat-GPT, a robust large language model. Using zero-shot, one-shot, and few-shot learning techniques, Chat-GPT annotates the unlabelled part of our dataset. These Chat-GPT-annotated pseudo-labels are then merged with manual annotations, facilitating the training of a multi-label emotion classification model via semi-supervised learning. Furthermore, within the context of our research, we present a new French tweet dataset, containing testimonials from people affected by an urban industrial incident. This dataset features 2,350 tweets, each manually annotated by three human annotators based on 8 pre-identified emotions. Benchmark results are presented for multi-label emotion classification models employing both fully supervised and semi-supervised approaches with pseudo-labeling. Our findings demonstrate the superiority of our approach for multi-label emotion classification over standard pseudo-labeling and an established baseline.
BibTeX
@inproceedings{malik2024pseudo,
  author = {Usman Malik and Simon Bernard and Alexandre Pauchet and Clément Chatelain and Romain Picot-Clémente and Jérôme Cortinovis},
  title = {{Pseudo-Labeling With Large Language Models for Multi-Label Emotion Classification of French Tweets}},
  booktitle = {IEEE Access},
  year = {2024},
  url = {https://openreview.net/forum?id=DCbPJtBxCO},
}
Figure