Abstract
Negative emotions expressed on social media platforms like Reddit are linguistically more complex than positive ones. This increased complexity contributes significantly to the performance gap observed in state-of-the-art Natural Language Processing (NLP) models. By analyzing the GoEmotions dataset through 12 linguistic metrics and Exploratory Factor Analysis (EFA), this study identifies three primary factors of complexity: Syntactic, Frequency-based, and Basic Lexical. Results indicate that negative emotions consistently score higher in syntactic complexity, leading to a "performance cliff" for transformer-based models like BERT and RoBERTa.
Introduction
Humans exhibit a natural "negativity bias," being more drawn to and affected by negative information. In the digital realm, this translates to a high volume of negative sentiment on social media. However, emotion detection models in NLP consistently struggle to identify negative emotions (e.g., grief, remorse) compared to positive ones (e.g., joy, gratitude). While previous research suggested data imbalance as the cause, this paper investigates whether the inherent linguistic complexity of negative expressions makes them fundamentally harder to model.
Methodology
The researchers employed a multi-stage approach to quantify linguistic difficulty:
- Metric Extraction: Twelve metrics were used, covering lexical richness (Type-Token Ratio), word frequency (DALE), and syntactic "tangledness" using measures like Dependency Locality Theory (DLT) and Incomplete Dependency Theory (IDT).
- Factor Analysis: Exploratory Factor Analysis (EFA) grouped these metrics into three distinct factors:
- Factor 1 (Syntactic): Includes IDT, DLT, Left-Embeddedness (LE), and Nested Noun Distance (NND).
- Factor 2 (Frequency-based): Focuses on rare vocabulary and log-likelihood.
- Factor 3 (Basic Lexical): Covers word length and polysyllabic word counts.
- Model Evaluation: BERT, RoBERTa, and SpanBERT were fine-tuned on the GoEmotions dataset and tested against varying levels of complexity.
Datasets
The study utilized GoEmotions, a dataset of 58,000 Reddit comments. The authors introduced CAMEO, an augmented version of GoEmotions annotated with the 12 complexity metrics, to facilitate further research into model failure modes.
Key Insights: The Complexity of Negativity
The analysis confirmed that negative emotions generate more complex text. Specifically, 9 out of 12 metrics showed significantly higher complexity for negative sentiments. Negative emotions are particularly prone to high syntactic complexity (Factor 1), characterized by convoluted sentence structures and long-distance word dependencies.
Results: The Performance Cliff
A staggering gap in AI performance was observed. BERT's micro-F1 score dropped from 0.4850 on "simple" positive emotions to just 0.2195 on "simple" negative emotions. All models performed worst on the most syntactically complex samples. Interestingly, SpanBERT did not outperform standard BERT on complex emotional language, despite its design for handling long-range dependencies.
Conclusion and Future Work
The findings suggest that "misery loves complexity" is a significant hurdle for emotion AI. While this study focused on English Reddit data, future work should investigate if these patterns hold across different languages and cultures. Additionally, developing models specifically architected to handle high syntactic loads may be necessary to bridge the performance gap in negative emotion detection.
Summary updated