Abstract
Recent advancements in Natural Language Processing (NLP) have led to high performance on emotion detection benchmarks. However, it remains unclear whether these models perform deep inferential reasoning or merely exploit superficial lexical cues. This paper introduces two "hard" evaluation sets, GoEmotions.v2 and CancerEmo.v2, created through an iterative model-annotator feedback loop. By rephrasing text to remove explicit emotion words, the researchers demonstrate a significant performance collapse in standard models, while Large Language Models (LLMs) exhibit greater resilience.
Introduction: The Lexical Shortcut Problem
Current NLP models often achieve high accuracy by latching onto explicit keywords. For instance, a model might correctly label a sentence as "sadness" simply because it contains the word "sad," without truly understanding the underlying context. This "shortcut learning" masks a lack of deep emotional understanding. This research addresses this gap by creating evaluation sets where these "dead giveaways" are removed, forcing models to rely on a deeper understanding of context and implicit meaning.
Methodology: The Model-Annotator Feedback Loop
To construct the "v2" datasets, the authors employed a human-in-the-loop strategy designed to "break" a baseline BERT model.
The process involved several key steps:
- Selection: Identifying examples from existing datasets (GoEmotions and CancerEmo) that a baseline BERT model correctly predicted.
- Iterative Rephrasing: Expert human annotators rephrased these examples to make the emotion implicit while maintaining the original sentiment.
- Stress Testing: Rephrased text was fed back to the BERT model. If the model still predicted correctly, annotators performed further rounds of rephrasing (up to four rounds, labeled R1 through R4).
- Categorization: Final texts were classified as Explicit (retaining emotion words), Implicit (context-only), or Misleading (using words typically associated with a different emotion, such as sarcasm).
Data Analysis and Model Performance
The iterative process successfully reduced the reliance on explicit cues. As shown in the figure below, the average number of explicit emotion words decreased significantly across the rephrasing rounds (R1 through R4) compared to back-translated (BT) baselines.
Key Findings
- Performance Collapse: As explicit keywords were removed, model performance plummeted. BERT's F1 score on GoEmotions dropped from 0.59 on the original set to just 0.05 on the R4 set.
- LLM Resilience: Larger models like OPT-IML (30B+) proved more robust, dropping from 0.65 to 0.15. This suggests that the broader world knowledge inherent in LLMs aids in interpreting implicit emotional context.
- The "Misleading" Challenge: The most difficult examples were those categorized as "misleading," where models were tricked by lexical cues that usually signal a different emotion.
- Robustness through Fine-Tuning: Training models on a small subset of rephrased examples (approx. 500) led to significant improvements on the hard test sets, indicating that models can be taught to look beyond simple keywords.
Conclusion and Future Directions
This research highlights a critical vulnerability in current emotion detection models: a heavy dependency on lexical crutches. While LLMs show promise in handling implicit context, there is still a wide gap between model performance and human-level emotional understanding. Future work will extend these "hard" benchmarks to non-English languages and diverse domains to further probe the limits of machine emotional intelligence.
Summary updated