Abstract
Traditional emotion recognition (ER) systems typically categorize text into discrete labels, such as "happy" or "sad." However, for sensitive applications like clinical telehealth or empathetic tutoring, understanding the specific linguistic evidence for an emotion is crucial. We present SEER (Span-based Emotion Evidence Retrieval), a benchmark designed to evaluate the ability of Large Language Models (LLMs) to pinpoint the exact words and phrases that convey emotional states. Our findings indicate that while LLMs show promise in single-sentence tasks, their performance degrades significantly in multi-sentence conversational contexts due to context-induced hallucinations and keyword fixation.
Introduction
The field of Emotion AI is shifting from simple classification to evidence-based retrieval. While identifying a speaker's overall sentiment is useful, high-stakes human-AI interaction requires transparency: why is the model labeling a specific segment as emotional? The SEER benchmark addresses this by focusing on "emotion evidence"—the specific linguistic spans that reveal internal states.
Problem Statement: Beyond Black-Box Classification
Existing emotion datasets often suffer from two major limitations. First, they lack granularity, providing only sentence-level or word-level labels that are either too coarse or too fragmented. Second, they frequently rely on scripted performances, which do not capture the nuances of spontaneous, real-world discourse.
As shown in the table above, SEER distinguishes itself by focusing specifically on emotion evidence retrieval within spontaneous data, filling a critical gap in the current landscape of emotion recognition benchmarks.
Methodology
The researchers evaluated 14 open-source LLMs, including the LLaMA, Qwen, Phi, and Gemma families, across two distinct tasks:
- Task 1 (Single-Sentence): Identifying evidence within a single non-neutral sentence.
- Task 2 (Multi-Sentence): Identifying evidence across a five-sentence passage where emotional states may shift or remain neutral.
To assess robustness, two prompting formats were utilized: Retrieve, where the model extracts the exact text spans, and Highlight, where the model reproduces the text with markers around the evidence.
Performance was measured using token-level F1-scores and Embedding Cosine Similarity (Sim) to account for the inherent subjectivity in defining span boundaries.
Datasets
The SEER benchmark comprises 1,200 real-world sentences sourced from two major speech corpora:
- MSP-Podcast: Spontaneous conversational speech from various podcasts.
- MuSE (Multimodal Stressed Emotion): Non-acted audiovisual monologues.
Transcripts were generated via Whisper, and a rigorous annotation pipeline was employed. Initial labels from GPT-4 were verified and refined by trained human annotators to establish high-quality "gold spans."
Key Insights and Results
The evaluation revealed several critical behaviors in modern LLMs:
The Context Collapse
While LLMs approach human-level performance on single sentences, their accuracy drops sharply in multi-sentence passages. This "context collapse" is primarily driven by false positives, where models incorrectly attribute emotion to neutral sentences simply because they appear near emotional ones.
Keyword Fixation vs. Linguistic Understanding
A recurring failure mode is "fixation," where models extract only a single salient word (e.g., "disgust") rather than the complete expressive phrase (e.g., "I would like to state my utter disgust"). This suggests that models often rely on simple keyword matching rather than a deep understanding of linguistic structure.
Model Performance and Prompt Sensitivity
- Top Performer: The Qwen3-32B model achieved the highest score on Task 1 (0.673 F1), outperforming even larger 70B-parameter models.
- Format Penalty: Models performed significantly worse in the "Highlight" format. Verbatim reproduction of long strings while inserting markers proved difficult, often leading to text hallucinations or unintended modifications.
- Chain-of-Thought (CoT): While CoT helped larger models in long contexts, it paradoxically hindered smaller models by causing them to over-focus on obvious keywords.
Conclusion and Future Work
The SEER benchmark demonstrates that while LLMs are becoming "emotionally literate" on the surface, pinpointing the nuances of emotional expression in complex dialogue remains a significant challenge. Future research should explore the integration of acoustic cues (e.g., tone, pauses) through "Audio-LLMs" to resolve textual ambiguities and expand the dataset to provide greater statistical power.
Summary updated