Systematic Evaluation of a Framework for Unsupervised Emotion Recognition for Narrative Text

Samira Zad, Mark FinlaysonACL 2020

TL;DR

This paper tackles the challenge of automatically recognizing emotions in narrative text, a crucial step for deeper narrative understanding. The authors reimplement and systematically evaluate a prior unsupervised framework, identifying key design choices and improving its performance by over 7% F1 score. They also highlight the limitations of existing emotion lexicons and suggest avenues for future research.

What Problem Does This Paper Address?

Understanding emotions expressed in text is vital for many applications, especially in computational narrative understanding. Narratives, by their nature, are rich with emotional content from characters and authors, and grasping these emotions is key to comprehending the story's meaning. However, accurately recognizing emotions in text is a complex task due to the subtle ways emotions are expressed, the ambiguity of language, and a general scarcity of high-quality labeled data. Previous work, like that by Kim et al. (2010), showed promise with unsupervised methods on narrative text, but the authors found significant reproducibility issues with that approach. This paper aims to address these reproducibility challenges and systematically evaluate and improve upon such unsupervised frameworks for narrative emotion recognition.

Methods

The core of this paper is the reimplementation and systematic evaluation of an unsupervised emotion recognition framework inspired by Kim et al. (2010). The proposed system follows a four-step process:

  1. Pre-processing: Sentences are tokenized and lemmatized using the CoreNLP library.
  2. Vector Space Modeling: A tf-idf vector is generated for each sentence, and standard vectors are created for each emotion class using an emotion lexicon.
  3. Noise Cancellation/Dimension Reduction: This is a key area of exploration. The authors compare three techniques:
  4. Non-negative Matrix Factorization (NMF): Used to extract latent features. This method involves additional steps of term selection, feature removal (based on emotional relevance), and vector space reconstruction.
  5. Principal Component Analysis (PCA): A standard technique for dimension reduction.
  6. Latent Dirichlet Allocation (LDA): Another common method for dimension reduction in NLP. The paper meticulously details the hyper-parameter tuning for these methods, particularly for NMF.

  7. Labeling: The transformed sentence vectors are compared to the emotion vectors using cosine similarity to assign an emotion label.

A significant contribution is the augmentation of the WordNet Affect (WNA) lexicon, addressing missing and incorrect emotion terms identified through careful inspection.

Datasets

  • Alm's corpus of fairy tales (Alm, 2008): This dataset, comprising 15,087 unique sentences from 176 fairy tales, was used for both training and evaluation. It was annotated with four emotion labels: joy, anger, fear, and sadness. The authors followed Kim et al. (2010) in using this dataset.
  • WordNet Affect (WNA): Used as the primary emotion lexicon. The authors augmented WNA by adding and removing terms, resulting in a final count of 4,048 lemma-POS pairs.

Key Insights and Ideas

  • Reproducibility is Paramount: The paper strongly emphasizes the difficulty in reproducing prior work and the necessity of detailed methodological descriptions and open-sourcing code/data for scientific progress.
  • NMF Outperforms PCA/LDA for this Task: While PCA and LDA are standard dimension reduction techniques, NMF, with careful hyper-parameter tuning and feature selection/removal, yielded superior results for emotion recognition in this narrative context.
  • Emotion Lexicons are a Bottleneck: The authors discovered numerous missing and incorrect emotion terms in the widely used WordNet Affect lexicon. This highlights that current lexicons are far from comprehensive and that improving them is crucial for better emotion recognition performance.
  • Context is King (and Hard to Capture): The paper acknowledges that even with improved lexicons and methods, capturing nuanced emotional expressions that rely heavily on context (e.g., "I'm over the moon!") remains a significant challenge.

Key Results

  • The reimplemented framework, after systematic evaluation and hyper-parameter tuning, achieved an overall F1 score of 0.809, which is 7.6 F1 points higher than the reported performance of Kim et al. (2010).
  • NMF was identified as the best-performing dimension reduction technique among NMF, PCA, and LDA.
  • The augmentation of the WNA lexicon led to a significant improvement in performance.
  • The paper provides detailed performance metrics (precision, recall, F1) for each emotion class and for different hyper-parameter settings of NMF, PCA, and LDA, allowing for a thorough understanding of their impact.

Limitations and Future Work

  • Lexicon Limitations: The authors acknowledge that even their augmented lexicon is likely not exhaustive and that a rigid lexicon-based approach has inherent limitations in capturing context-dependent emotions.
  • Context Dependency: The inherent difficulty of capturing emotions that rely heavily on context, rather than explicit emotion words, remains a significant challenge.
  • Data Scarcity and Quality: The paper points to the general lack of high-quality annotated data, with existing corpora often suffering from inconsistent annotations and low inter-annotator agreement.

Future work directions suggested include:

  • Reconciling emotion lexicons with context-dependent emotion detection.
  • Evaluating bag-of-words multi-layer perceptrons for emotion extraction.
  • Exploring multi-label prediction approaches.
  • Investigating the impact of text unit size (sentence vs. paragraph vs. story) on sentiment analysis.
  • Careful annotation of larger corpora and further augmentation of emotion lexicons.

Summary updated

BibTeX
@inproceedings{zad2020systematic,
  author = {Samira Zad and Mark Finlayson},
  title = {{Systematic Evaluation of a Framework for Unsupervised Emotion Recognition for Narrative Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure