Misery Loves Complexity: Exploring Linguistic Complexity in the Context of Emotion Detection

Pranaydeep Singh, Luna De Bruyne, Orphée De Clercq, Els LefeverEMNLP 2023 · DOI

Abstract

Negative emotions expressed on social media platforms like Reddit are linguistically more complex than positive ones. This increased complexity contributes significantly to the performance gap observed in state-of-the-art Natural Language Processing (NLP) models. By analyzing the GoEmotions dataset through 12 linguistic metrics and Exploratory Factor Analysis (EFA), this study identifies three primary factors of complexity: Syntactic, Frequency-based, and Basic Lexical. Results indicate that negative emotions consistently score higher in syntactic complexity, leading to a "performance cliff" for transformer-based models like BERT and RoBERTa.

Introduction

Humans exhibit a natural "negativity bias," being more drawn to and affected by negative information. In the digital realm, this translates to a high volume of negative sentiment on social media. However, emotion detection models in NLP consistently struggle to identify negative emotions (e.g., grief, remorse) compared to positive ones (e.g., joy, gratitude). While previous research suggested data imbalance as the cause, this paper investigates whether the inherent linguistic complexity of negative expressions makes them fundamentally harder to model.

Methodology

The researchers employed a multi-stage approach to quantify linguistic difficulty:

  1. Metric Extraction: Twelve metrics were used, covering lexical richness (Type-Token Ratio), word frequency (DALE), and syntactic "tangledness" using measures like Dependency Locality Theory (DLT) and Incomplete Dependency Theory (IDT).
  2. Factor Analysis: Exploratory Factor Analysis (EFA) grouped these metrics into three distinct factors:
  3. Factor 1 (Syntactic): Includes IDT, DLT, Left-Embeddedness (LE), and Nested Noun Distance (NND).
  4. Factor 2 (Frequency-based): Focuses on rare vocabulary and log-likelihood.
  5. Factor 3 (Basic Lexical): Covers word length and polysyllabic word counts.
Factor loadings for the three identified linguistic factors, highlighting the dominance of syntactic metrics in Factor 1.
  1. Model Evaluation: BERT, RoBERTa, and SpanBERT were fine-tuned on the GoEmotions dataset and tested against varying levels of complexity.

Datasets

The study utilized GoEmotions, a dataset of 58,000 Reddit comments. The authors introduced CAMEO, an augmented version of GoEmotions annotated with the 12 complexity metrics, to facilitate further research into model failure modes.

Key Insights: The Complexity of Negativity

The analysis confirmed that negative emotions generate more complex text. Specifically, 9 out of 12 metrics showed significantly higher complexity for negative sentiments. Negative emotions are particularly prone to high syntactic complexity (Factor 1), characterized by convoluted sentence structures and long-distance word dependencies.

Comparison of Incomplete Dependency Theory (IDT) scores, showing that negative emotions exhibit higher syntactic complexity than positive or neutral ones.

Results: The Performance Cliff

A staggering gap in AI performance was observed. BERT's micro-F1 score dropped from 0.4850 on "simple" positive emotions to just 0.2195 on "simple" negative emotions. All models performed worst on the most syntactically complex samples. Interestingly, SpanBERT did not outperform standard BERT on complex emotional language, despite its design for handling long-range dependencies.

Conclusion and Future Work

The findings suggest that "misery loves complexity" is a significant hurdle for emotion AI. While this study focused on English Reddit data, future work should investigate if these patterns hold across different languages and cultures. Additionally, developing models specifically architected to handle high syntactic loads may be necessary to bridge the performance gap in negative emotion detection.

Summary updated

BibTeX
@inproceedings{singh2023misery,
  author = {Pranaydeep Singh and Luna De Bruyne and Orphée De Clercq and Els Lefever},
  title = {{Misery Loves Complexity: Exploring Linguistic Complexity in the Context of Emotion Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.857},
}
Figure