Fearful Falcons and Angry Llamas: Emotion Category Annotations of Arguments by Humans and LLMs

Lynn Greschner, Roman KlingerNAACL 2025

TL;DR

This paper introduces Emo-DeFaBel, the first corpus of discrete emotion categories in German arguments, and evaluates how well Large Language Models (LLMs) can automatically annotate these emotions. The study reveals that while LLMs show high recall for negative emotions, their precision is low, indicating a bias. Understanding discrete emotions in arguments is crucial for better analysis and persuasion modeling.

What Problem Does This Paper Address?

Arguments are known to evoke emotions, which in turn influence their effectiveness. While previous research has focused on binary emotionality (whether an argument is emotional or not) or emotion intensity, this paper highlights a gap: the importance of discrete emotion categories (like anger, joy, fear) in understanding how arguments affect persuasion and cognitive processes. Existing Natural Language Processing (NLP) research has largely overlooked this, creating a mismatch with psychological and argumentation theory that emphasizes the role of specific emotion categories. This paper aims to bridge this gap by providing a dataset and evaluating LLM capabilities in this nuanced area.

Methods

The authors employed a multi-faceted approach:

  • Human Annotation: They crowdsourced subjective annotations of emotion categories for German argumentative texts. Participants were asked to identify the most present emotion evoked by an argument from a predefined set of basic emotions, cognitive evaluations, and self-directed states.
  • LLM Evaluation: They tested three instruction-tuned LLMs (Falcon-7b-instruct, Llama-3.1-8B-instruct, and GPT-40-mini) on emotion annotation.
  • Prompting Strategies: Three prompting strategies were compared:
  • Zero-shot: Providing only instructions.
  • One-shot: Providing instructions plus one example.
  • Chain-of-Thought (CoT): Encouraging the LLM to generate a reasoning process before the final answer.
  • Emotion Domain Settings: They explored three ways of defining the output space for LLMs:
  • Binary: Whether the argument contains any emotion.
  • Closed-domain: Predicting from a predefined set of emotion labels.
  • Open-domain: Allowing the LLM to predict any emotion label.
  • Evaluation Metrics: They used both strict (matching majority vote) and relaxed (matching any human annotation) evaluation modes to account for the subjectivity of emotion annotation.

Datasets

  • DeFaBel Corpus: The study used the DeFaBel corpus as a basis, which contains argumentative German texts generated to support statements from the TruthfulQA dataset.
  • Emo-DeFaBel Corpus: This is a newly created corpus by the authors, consisting of 300 arguments from DeFaBel that were manually annotated with discrete emotion categories. This is presented as the first argumentative corpus labeled with emotion categories.

Key Insights and Ideas

  • Discrete Emotions Matter: The study strongly suggests that discrete emotion categories, not just binary emotionality, are crucial for understanding argument effectiveness.
  • LLM Bias Towards Negativity: LLMs exhibit a significant bias towards predicting negative emotions, particularly anger and fear, showing high recall but low precision. This indicates they are good at spotting negative sentiment but struggle to differentiate specific negative emotions or correctly identify positive ones.
  • Subjectivity is Key: Emotion annotation is inherently subjective. The authors found substantial disagreement among human annotators, which is reflected in the LLM performance. The relaxed evaluation metric, which accounts for this subjectivity, often yielded better results.
  • Prompting and Domain Influence: While the study explored different prompting strategies and emotion domain settings, the performance differences were often minor, suggesting that the inherent subjectivity of the task might be a larger factor than the specific LLM or prompting technique. However, inferring binary emotionality from discrete emotion labels generally performed better than directly asking for binary labels.
  • Stance and Emotion Correlation: There's a potential correlation between annotator stance towards an argument and the emotion they assign, particularly for surprise and uncertainty.

Key Results

  • Human Annotation: Positive emotions like joy and pride were correlated with higher perceived convincingness, while anger was negatively correlated.
  • LLM Performance:
    • LLMs generally showed high recall but low precision for predicting anger and fear, indicating a bias.
  • GPT performed best in the relaxed evaluation setting for discrete emotion predictions.
  • Falcon performed best in the binary emotion prediction setting.
  • The performance differences across prompting strategies (zero-shot, one-shot, CoT) and emotion domains (binary, closed, open) were often marginal, especially in the strict evaluation.
  • Models struggled significantly with predicting shame and guilt.
  • Inter-Annotator Agreement: Agreement among human annotators was generally low, especially for less frequent emotions, highlighting the task's subjectivity. The value for Krippendorff's alpha was notably low (0.04).

Limitations and Future Work

  • Subjectivity of Annotation: The inherent subjectivity of emotion annotation is a major limitation. The authors acknowledge that LLMs, without explicit persona mimicry, only partially address this.
  • Corpus Limitations: The Emo-DeFaBel corpus is based on German arguments generated in a controlled manner. Findings might not generalize to online discourse or arguments in other languages. The corpus size is also noted as comparably small.
  • LLM Generalization: The study used three LLMs; results might differ with other models. Predictions can also vary between runs.
  • Future Work:
    • Investigate fine-tuning or prompt optimization for better emotion prediction.
  • Improve precision for individual emotion classes.
  • Conduct a fine-grained analysis of argumentative structure and quality to understand its impact on emotion.
  • Expand the study with a perspectivist approach (e.g., considering annotator demographics and stances).
  • Explore arguments in discourse rather than isolated statements.
  • Investigate LLM biases more deeply and develop methods to mitigate them.

Summary updated

BibTeX
@inproceedings{greschner2025fearful,
  author = {Lynn Greschner and Roman Klinger},
  title = {{Fearful Falcons and Angry Llamas: Emotion Category Annotations of Arguments by Humans and LLMs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure