eMLM: A New Pre-training Objective for Emotion Related Tasks

Tiberiu Sosea, Cornelia Caragea

Abstract

Standard Masked Language Modeling (MLM) in BERT treats all tokens with equal importance during pre-training. However, for affective computing tasks, emotionally charged words often carry disproportionate semantic weight. This paper proposes Emotion Masked Language Modeling (eMLM), a pre-training strategy that biases the model to focus on emotionally rich words by masking them more frequently. Our results demonstrate that eMLM improves performance across sentiment and emotion detection tasks while significantly enhancing model robustness against adversarial perturbations.

Introduction

Pre-trained language models like BERT have revolutionized NLP through task-agnostic objectives like MLM, where 15% of tokens are randomly masked. While effective for general language understanding, this uniform approach may be suboptimal for specialized domains like sentiment analysis or emotion detection. In these tasks, words such as "ecstatic" or "devastated" are far more critical to the final classification than functional words like "the" or "was."

The authors argue that by prioritizing the prediction of emotionally rich words during the computationally intensive pre-training phase, models can develop a more nuanced understanding of emotional context.

Methodology

The core of this work is the Emotion Masked Language Modeling (eMLM) objective. Unlike standard MLM, which uses a flat 15% probability for all tokens, eMLM employs a non-uniform masking strategy:

  1. Lexicon Integration: The EmoLex lexicon is used to identify "emotionally rich" words within the pre-training corpus (Wikipedia and BookCorpus).
  2. Biased Masking: Words found in the lexicon are assigned a higher masking probability ($k$). Empirical testing identified $k=0.5$ as the optimal value.
  3. Probability Balancing: To maintain a total masking rate of 15% for fair comparison with vanilla BERT, the masking probability for non-emotional words is proportionally reduced.
Comparison of masking probabilities between standard MLM and eMLM.

The authors trained eMLM-BERT from scratch to ensure these emotional biases were deeply integrated into the base representations rather than just fine-tuned.

Experimental Setup

The model was evaluated on four major benchmarks:

  • Sentiment Analysis: Stanford Sentiment Treebank (SST-2 for binary and SST-5 for fine-grained classification).
  • Emotion Detection: GoEmotions (27 categories from Reddit) and CancerEmo (8 basic emotions from health communities).
  • Lexicons: EmoLex was the primary source for identifying ~6,000 emotion-associated words and ~5,500 sentiment-associated words.

Results and Discussion

eMLM consistently outperformed standard BERT across all evaluated tasks, with an average improvement of 1.2% F1.

Performance comparison on SST-2 and SST-5 sentiment analysis benchmarks.

Key Insights

  • Granularity and Transfer: Two versions were tested: eMLM (S) for general sentiment and eMLM (E) for specific emotions. Interestingly, eMLM (E) performed better even on general sentiment tasks, suggesting that learning specific emotional nuances provides a superior foundation for broader sentiment understanding.
  • Enhanced Robustness: A common concern with biased training is that the model might become overly reliant on specific keywords. However, an "EmoWord" perturbation test (removing emotion words) revealed that eMLM was twice as robust as vanilla BERT. The success rate of such adversarial attacks was only 5.4% for eMLM, compared to 9.8% for BERT. This indicates that eMLM learns the contextual patterns surrounding emotion, not just the words themselves.
  • The $k$ Factor Sensitivity: Performance peaked at $k=0.5$. Higher values (e.g., 0.9) led to degradation, likely because the model lost the necessary surrounding context to accurately predict the masked tokens.

Conclusion

eMLM offers a simple yet effective modification to the BERT pre-training objective for affective computing. By focusing on emotionally salient tokens, the model achieves higher accuracy and greater robustness. Future work may explore incorporating emotion intensity into the masking probability and extending the methodology to multi-lingual settings.

Summary updated

BibTeX
@inproceedings{sosea2021emlm,
  author = {Tiberiu Sosea and Cornelia Caragea},
  title = {{eMLM: A New Pre-training Objective for Emotion Related Tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.38},
}
Figure