zhouyijiang1 at SemEval-2025 Task 11: A Multi-tag Detection Method based on Pre-training Language Models

Zhou Jiang, Dengtao ZhangACL 2025

TL;DR

This paper presents a BERT-based approach for multi-tag emotion detection in social media text, achieving a competitive F1 score of 0.70. The method leverages BERT's deep contextual understanding and introduces a novel asymmetric loss function to better handle imbalanced multi-label classification scenarios.

What Problem Does This Paper Address?

The paper tackles the challenge of "Bridging the text-based emotion detection gap" in SemEval-2025 Task 11. Emotion detection, a crucial aspect of Natural Language Processing (NLP) and sentiment analysis, aims to automatically identify and understand the emotional nuances in text. While traditional methods have relied on template matching or shallow machine learning, they struggle with the complexities of semantic representation and capturing emotional interactions in nuanced contexts like irony. More advanced deep learning models like RNNs and CNNs have improved context awareness but still face limitations in modeling long-distance emotional dependencies. This work specifically addresses the need for accurate multi-tag emotion detection, where a single text can express multiple emotions simultaneously, a common scenario in social media.

Methods

The core of the proposed method is the BERT-base-uncased pre-trained language model. BERT's Transformer architecture allows for deep bidirectional context coding, enabling a richer understanding of text semantics and potential emotions.

Key methodological components include:

  • Input Representation: The model concatenates text fragments with special tokens [CLS] and [SEP] to represent consecutive sentences, allowing BERT to process them as a single input. Token, Segment, and Position Embeddings are utilized.
  • Feature Extraction: BERT encodes the text, and the representation from the [CLS] token is used as the global semantic representation.
  • Gating Mechanism: A gating mechanism is employed to fuse the [CLS] token's representation with other contextual features.
  • Multi-label Classification Output: A three-layer fully connected network with ReLU activation is used, with the output layer employing sigmoid activation to generate multi-label probabilities.
  • Asymmetric Loss Function: To address the challenge of imbalanced multi-label classification (where positive signals can be overwhelmed by frequent negative classes), the authors introduce a novel sequential loss function. This function uses a linear transformation to create a symmetric solution space for positive and negative classes, weakening the influence of negative labels. A hard mask mechanism further separates positive and negative classes for loss calculation. This is a significant departure from traditional binary cross-entropy.
  • Training Strategy: The training process includes data loading optimization, adaptive optimizers, dynamic learning rate scheduling (StepLR), and an early stopping mechanism to prevent overfitting. The first 8 BERT layers are frozen, and higher layers are fine-tuned.

Datasets

The study uses the English subset of the SemEval-2025 Task 11 benchmark dataset.

  • Training Set: 15,000 manually annotated social media short texts.
  • Validation Set: 3,000 samples.
  • Test Set: 2,000 samples.

The dataset covers fine-grained annotations for five fundamental emotions: anger, fear, joy, sadness, and surprise. A stratified random sampling strategy was employed to ensure proportional representation of each emotional category and mitigate data bias.

Key Insights and Ideas

  • BERT's Efficacy for Multi-tag Emotion: The paper demonstrates that BERT, with its deep contextual understanding, is highly effective for multi-tag emotion detection, outperforming previous approaches that struggled with nuanced emotional expression.
  • The Importance of Handling Label Imbalance: The introduction of an asymmetric loss function highlights a critical challenge in multi-label classification. By explicitly addressing the dominance of negative classes, the model can better learn to identify positive emotion tags, especially in sparse scenarios.
  • Cross-Domain Emotion Recognition: The achieved F1 score of 0.70 suggests the effectiveness of their transfer learning framework for cross-domain emotion recognition, implying the model can generalize well to unseen emotional expressions.

Key Results

The proposed BERT-based model achieved a Macro-F1 score of 0.70 on the SemEval-2025 Task 11 dataset. The authors note that the model performs stably and well in terms of accuracy and recall, indicating a good balance in predicting both positive and negative class tags. The average F1 value being close to 0.70 suggests the model can predict different labels with comparable effectiveness.

Limitations and Future Work

The authors acknowledge several limitations:

  • Reliance on [CLS] Token: The model's architecture relies on a single linear layer atop BERT's [CLS] embeddings, potentially overlooking inter-label dependencies.
  • Fixed Truncation Length: The fixed truncation length of 128 tokens might lead to the discarding of critical emotional cues in longer dialogues.

Future work directions suggested include:

  • Dynamic Length Adaptation: Exploring methods to handle variable-length inputs more effectively.
  • Hierarchical Label Interaction Modules: Incorporating modules that explicitly model relationships between different emotion labels.
  • Focus Technology Integration: Investigating the integration of focus mechanisms to improve BERT's attention on core terms within the text.
  • Dynamic Context Perception and Local-Global Feature Fusion: Developing strategies to better decouple emotional conflicts and semantic meanings.
  • Lightweight Extraction or Mixture of Experts (MoE): Exploring more efficient model architectures for practical applications.

Summary updated

BibTeX
@inproceedings{jiang2025zhouyijiang,
  author = {Zhou Jiang and Dengtao Zhang},
  title = {{zhouyijiang1 at SemEval-2025 Task 11: A Multi-tag Detection Method based on Pre-training Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure