“Going to a trap house” conveys more fear than “Going to a mall”: Benchmarking Emotion Context Sensitivity for LLMs

Eojin Jeon, Mingyu Lee, Sangyun Kim, Junho Kim, Wanzee Cho, Tae-Eui Kam, SangKeun LeeEMNLP 2025

TL;DR

This paper introduces TRACE, a novel benchmark designed to evaluate how well Large Language Models (LLMs) understand emotion context sensitivity – the ability to adjust emotional responses based on situational nuances. Despite significant progress, current LLMs lag considerably behind human performance in this crucial aspect of emotional intelligence, highlighting a key area for future AI development.

What Problem Does This Paper Address?

Human emotional intelligence is deeply intertwined with context. We don't react to the same event identically if the circumstances change. For instance, an invitation to "come with me" can elicit joy if the destination is a mall but fear if it's a trap house. As LLMs become more integrated into social interactions, understanding and replicating this context-sensitive emotional reasoning is paramount for creating truly intelligent and empathetic AI. Existing benchmarks primarily focus on general emotion recognition, failing to specifically assess how LLMs handle the subtle shifts in emotion driven by contextual variations. This paper aims to fill that gap by proposing a benchmark that isolates and tests this specific capability.

Methods

The core of this work is the TRACE benchmark, which comprises two complementary tests:

  • Sensitivity Test: This test assesses whether LLMs can correctly identify emotional shifts when the context is deliberately changed to elicit different emotions.
  • Robustness Test: This test evaluates whether LLMs can maintain stable emotion predictions when the context changes are emotionally irrelevant.

To construct TRACE, the authors followed a rigorous data collection pipeline:

  1. Scenario Classification: Crowdworkers identified "pivot characters" in scenarios and determined if context changes would likely alter their emotions.
  2. Context Modification: Scenarios were modified by altering time, place, or agent details to create pairs for either the sensitivity or robustness test.
  3. Emotion Labeling: A detailed taxonomy of 64 emotions, expanded from Plutchik's Wheel, was used for consistent annotation.
  4. Distractor Addition: Adversarial distractors were generated using LLMs to create challenging multiple-choice questions.
  5. Validation: A two-round validation process by expert crowdworkers ensured data quality.

The benchmark consists of 1,626 scenarios, systematically curated to isolate the impact of different contextual dimensions.

Datasets

The primary contribution is the introduction of the TRACE benchmark. This benchmark is novel in its focus on emotion context sensitivity and its design to disentangle the effects of time, place, and agent variations. Existing benchmarks are mentioned as primarily focusing on general emotion recognition without this specific contextual focus.

Key Insights and Ideas

  • LLMs Still Struggle with Nuance: Even the most advanced LLMs significantly underperform humans in understanding emotion context sensitivity. This suggests that current models, despite their impressive capabilities in other areas, lack a fundamental grasp of how context shapes human emotions.
  • Robustness is a Major Hurdle: The robustness test revealed that LLMs often struggle to ignore irrelevant contextual changes, sometimes overestimating their emotional impact. This indicates a difficulty in selectively reasoning about what is emotionally pertinent.
  • Time and Agent Contexts are Particularly Challenging: LLMs showed the lowest performance when context changes involved time and agent. This suggests a need for deeper research into how AI can better model temporal perception and the complex interplay between emotions and social agents.
  • Over-reliance on Keywords: Qualitative analysis revealed that LLMs tend to over-rely on emotionally salient keywords, leading to correct predictions when the context aligns with stereotypes but incorrect ones when it deviates.

Key Results

  • The best-performing LLM (o1) lagged behind human performance by approximately 20% in the sensitivity test and 15% in the robustness test.
  • Even with advanced prompting strategies like Chain-of-Thought (CoT), improvements were marginal, indicating inherent limitations rather than just prompting issues.
  • Error analysis showed that LLMs tend to underestimate or misjudge how contextual changes affect emotional states (48% underestimates, 32% misinterpretations).
  • Capturing emotional shifts due to time context was the most challenging for both humans and LLMs.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Individual Differences: TRACE primarily isolates single context dimensions, potentially missing how multiple factors or individual differences (like cultural background) interact.
  • Textual Modality Only: The benchmark focuses solely on text, overlooking the multimodal nature of emotional cues (visual, auditory).
  • Focus on Affective Experience: TRACE assesses the inference of emotional states but not emotional expression or regulation.
  • US-Centric Scenarios: The dataset may reflect emotional experiences more common to US residents, limiting generalizability.

Future work could address these by incorporating multimodal data, exploring emotional expression and regulation, and collecting scenarios from diverse populations. The authors hope TRACE will spur further research into context-aware emotional reasoning in AI.

Summary updated

BibTeX
@inproceedings{jeon2025going,
  author = {Eojin Jeon and Mingyu Lee and Sangyun Kim and Junho Kim and Wanzee Cho and Tae-Eui Kam and SangKeun Lee},
  title = {{“Going to a trap house” conveys more fear than “Going to a mall”: Benchmarking Emotion Context Sensitivity for LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure