wangkongqiang at SemEval-2025 Task 11:Bridging the Gap in Text-Based Emotion Detection

Wang KongqiangACL 2025

TL;DR

This paper tackles the challenge of accurately detecting emotions in text, a task complicated by the nuanced and often subtle ways humans express feelings. The authors propose an ensemble learning approach, combining multiple pre-trained language models with a novel weighted voting system, to achieve strong performance on the SemEval-2025 Task 11 for multi-label emotion detection.

What Problem Does This Paper Address?

Emotion detection from text is a complex problem because human emotions are multifaceted and can be expressed indirectly. While we all experience and express emotions daily, articulating them precisely can be difficult, and people vary significantly in how they perceive and convey feelings. This variability makes it challenging to definitively identify an emotion solely from spoken or written words. The SemEval-2025 Task 11 specifically aims to bridge this gap by requiring systems to predict one or more emotions (joy, sadness, fear, anger, surprise, disgust) from a given text snippet. This multi-label nature adds another layer of complexity, as a single piece of text might evoke multiple emotions simultaneously.

Methods

The core of the proposed solution is an ensemble learning approach that leverages multiple pre-trained language models. The authors experimented with four prominent models: google-bert, FacebookAI-roberta, dccuchile-bert, and distilbert-multi.

A key innovation is their weighted voting system for combining the predictions of these individual models. Unlike traditional voting systems that might use a simple arithmetic mean, this approach assigns weights to each model's prediction based on its performance (specifically, its F1-score or RMSE) on a validation set. This allows models that perform better on the task to have a greater influence on the final prediction, while still incorporating the insights from other models.

The implementation involved several steps:

  1. Data Preparation: Ensuring the data was in the correct format for the simpletransformers Python library.
  2. Model Definition: Gathering the different transformer models to be evaluated.
  3. Individual Model Training: Training each pre-trained model separately on the entire training dataset.
  4. Ensemble Definition: Creating ensembles of these trained models.
  5. Evaluation: Evaluating each individual transformer and then each ensemble on a validation set to determine their metrics (accuracy, F1-score, precision, recall).
  6. Weighted Voting: Implementing a voting function that can either be unweighted or weighted based on normalized F1-scores.
  7. Best Ensemble Selection: Identifying the ensemble that achieved the highest F1-score on the validation set.
  8. Test Set Prediction: Using the selected best ensemble to predict labels for the test set.

Datasets

The paper primarily utilizes datasets for the SemEval-2025 Task 11, specifically focusing on Track A: Multi-label Emotion Detection. The training data consists of annotated tweets. The authors also mention using a development set for evaluation and a test set for final performance assessment. Table 1 provides a summary of the training set's text length distribution and the count of each emotion label. Figure 1 and Figure 2 further illustrate the length and quantity distribution of training text data and the distribution of text sizes for each class, respectively. Table 5 indicates that the dataset is supported by SemEval-2025 Task 11 on Track A and is based on raw data, with no external or additional corpora used.

Key Insights and Ideas

  • Ensemble Power: Combining multiple pre-trained language models significantly improves performance over single models, a common but effective strategy in NLP.
  • Weighted Voting Matters: A novel weighted voting system, where model contributions are based on their individual performance (F1-score), proved more effective than a simple averaging approach. This highlights the importance of dynamically adjusting the influence of each model based on its demonstrated capability.
  • Hyperparameter Sensitivity: The study involved experimenting with different hyperparameters (optimizer, learning rate) and repeating runs to find optimal configurations, underscoring the need for careful tuning in complex NLP tasks.
  • Data Characteristics Inform Design: The analysis of text length distribution (e.g., 75% of tweets having no more than 25 words) can inform decisions about network architecture or sentence length limits in future work.

Key Results

The proposed system achieved competitive results on the SemEval-2025 Task 11.

  • On the development set, the system achieved an Emotion Macro F1 Score of 0.6998 and an Emotion Micro F1 Score of 0.7374.
  • The paper states that the submission achieved a "good ranking place in the test set."
  • Specific scores for individual emotions on the dev set are provided in Table 3, with Joy scoring the highest at 0.7794 Macro F1.
  • Table 4 shows the test set results, with Fear achieving the highest individual emotion score at 0.8152 Macro F1.

Limitations and Future Work

  • English-Centric: The experiments were exclusively conducted on English language datasets. This is a significant limitation, as the nuances of emotion expression can vary greatly across languages.
  • Low-Resource Languages: The authors express regret that they could not explore other Asian and African languages due to data and pre-trained model availability.
  • Future Directions: Future research should focus on low-resource languages, including data collection, annotation, and the development of pre-training models tailored to these languages. This would contribute to a more globally applicable understanding of emotion detection.

Summary updated

BibTeX
@inproceedings{kongqiang2025wangkongqiang,
  author = {Wang Kongqiang},
  title = {{wangkongqiang at SemEval-2025 Task 11:Bridging the Gap in Text-Based Emotion Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure