Who Feels What and Why? Annotation of a Literature Corpus with Semantic Roles of Emotions

Evgeny Kim, Roman KlingerCOLING 2018

TL;DR

This paper introduces REMAN, a novel corpus for analyzing emotions in literature that goes beyond simply identifying emotions to also capture who feels the emotion and why. They demonstrate that traditional emotion analysis methods are insufficient and that joint modeling of emotions with their semantic roles (experiencer, cause, target) significantly improves prediction performance.

What Problem Does This Paper Address?

Current approaches to emotion analysis in literature often focus solely on detecting the presence of an emotion. This is a significant oversimplification, as it ignores crucial contextual information: who is experiencing the emotion and what triggered it. Emotions are fundamental to character interactions and plot development in narratives. However, existing datasets for literary emotion analysis lack annotations for these semantic roles, making it difficult to build models that truly understand the nuances of emotional expression in text. This paper aims to fill this gap by creating a corpus that annotates not just emotions but also their experiencers, causes, and targets.

Methods

The authors introduce the REMAN (Relational EMotion ANnotation) corpus, built from Project Gutenberg texts. The annotation process is multi-step and involves:

  1. Emotion Phrase Annotation: Identifying spans of text that express an emotion, including implicit or modified expressions.
  2. Emotion Phrase Aggregation: Semi-automatically merging annotations from multiple annotators.
  3. Relation Annotation: Identifying entities (characters, events, etc.) that act as experiencers, causes, or targets of the annotated emotions, and establishing these relationships.
  4. Relation Annotation Aggregation and Adjudication: A manual expert step to refine relation annotations.

They also propose baseline models for predicting emotion phrases and their associated roles. These models include:

  • Dictionary-based and Bag-of-Words (BOW) classifiers: For coarse-grained emotion classification.
  • Conditional Random Fields (CRF) and Bidirectional LSTM-CRF (biLSTM-CRF): For sequence labeling tasks, predicting emotion phrases and their roles.

A key methodological contribution is the evaluation of joint modeling, where models are trained to predict emotions and roles simultaneously, rather than as separate tasks.

Datasets

  • REMAN Corpus: A newly introduced dataset consisting of 200 books sampled from Project Gutenberg, focusing on fiction written by authors born after 1800. The corpus contains approximately 1115 sentences with emotion annotations, further annotated with experiencers, causes, and targets.
  • NRC Emotion Lexicon: Used as a feature in some baseline models.

Key Insights and Ideas

  • Beyond Emotion Detection: Simply identifying an emotion word (e.g., "afraid") is insufficient. Implicit emotions (e.g., "tense and frightened") and indirect expressions are crucial for a comprehensive understanding.
  • The Importance of Context: Understanding who feels what and why is essential for a deeper analysis of literary texts. Ignoring these roles leads to an overgeneralized interpretation of results.
  • Limitations of Lexicons: Emotion lexicons alone struggle to capture the high variability of emotion expressions and their associated roles in natural language.
  • Joint Modeling is Key: Predicting emotions and their semantic roles (experiencer, cause, target) together significantly improves performance compared to treating these as independent tasks. This highlights the strong complementarity between these elements.
  • Subjectivity and IAA: Annotating emotions and their roles is inherently subjective, leading to lower inter-annotator agreement (IAA) compared to simpler tasks. However, the multi-step annotation process did improve agreement for roles.

Key Results

  • Baseline Performance: Dictionary-based and BOW approaches for emotion classification showed mediocre performance, with high recall but low precision.
  • Sequence Labeling: biLSTM-CRF models achieved the highest precision and F1 scores for emotion category prediction (strict F1 = 43, fuzzy F1 = 48), outperforming dictionaries and BOW.
  • Role Prediction Challenges: Recall for role prediction (experiencer, cause, target) was generally low, particularly for experiencers, likely due to their expression through pronouns and the limited context provided (sentence triples).
  • Joint Modeling Benefits: Experiment 3 showed that joint modeling significantly boosted performance:
  • For emotion prediction, F1 increased by 5 pp (strict) and 9 pp (fuzzy) when the model was informed about the experiencer.
  • For experiencer prediction, F1 increased by 20 pp (strict) and 22 pp (fuzzy) when the model was informed about the emotion.

Limitations and Future Work

  • Limited Context: The use of sentence triples, while a trade-off for corpus size, may not always provide sufficient context for accurate annotation, especially for complex literary texts with allusions and metonymies.
  • Subjectivity: The inherent subjectivity of emotion annotation makes achieving perfect inter-annotator agreement challenging.
  • Future Work:
    • Investigate methods for pre-selecting more relevant context for annotation.
  • Develop more sophisticated joint models for structured prediction of emotions and their roles.
  • Explore how to better handle the ambiguity and variability in emotion expressions and role assignments.
  • The corpus is intended to facilitate future research in fine-grained emotion analysis and support qualitative literary studies and digital humanities.

Summary updated

BibTeX
@inproceedings{kim2018who,
  author = {Evgeny Kim and Roman Klinger},
  title = {{Who Feels What and Why? Annotation of a Literature Corpus with Semantic Roles of Emotions}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure