DENS: A Dataset for Multi-class Emotion Analysis

Chen Liu, Muhammad Osama, Anderson de Andrade

Introduction

Traditional emotion analysis in Natural Language Processing (NLP) has largely focused on short-form text, such as tweets or product reviews, often limited to binary sentiment classification. However, human narratives are far more complex, requiring an emotional spectrum that spans beyond simple positive or negative labels. To address this gap, researchers from Wattpad have introduced DENS (Dataset for Emotions of Narrative Sequences), a new benchmark designed for multi-class emotion analysis in long-form fiction.

DENS challenges models to identify complex emotions across multi-sentence passages, moving the field toward a more nuanced understanding of narrative journeys. By providing a large-scale, labeled dataset of both classic and modern literature, DENS serves as a rigorous testing ground for state-of-the-art NLP architectures.

The DENS Dataset

The DENS dataset consists of 9,710 passages, each ranging from 40 to 200 tokens. The data is sourced from two distinct areas: classic literature from Project Gutenberg (1,520 passages) and modern narratives from Wattpad. This breadth ensures the dataset captures diverse linguistic styles and contemporary themes, including Mystery, Fantasy, and Romance.

Selected examples from the DENS dataset showing passages and their associated emotion labels.

The final taxonomy includes nine labels: Joy, Sadness, Anger, Fear, Anticipation, Surprise, Love, Disgust, and Neutral. This selection reflects the specific emotional nuances found in storytelling, prioritizing categories that human readers can consistently identify within a narrative flow.

Methodology

The construction of DENS involved a multi-stage pipeline:

  1. Framework Selection: The authors adapted Plutchik’s Wheel of Emotions for narrative contexts, replacing "Trust" with "Love" and adding "Neutral" after finding that readers struggled to distinguish trust as a standalone emotion in fiction.
  2. Passage Selection: Lexicon-based pruning was used to prioritize passages containing emotionally evocative but ambiguous words, ensuring the dataset remained focused on non-trivial emotional content.
  3. Annotation: Amazon Mechanical Turk "Master" annotators labeled each passage. A majority agreement was reached for 73.5% of the data, with in-house experts resolving any remaining ties.
  4. Benchmarking: The researchers evaluated a spectrum of models, ranging from traditional Bag-of-Words (TF-IDF + SVM) to advanced deep learning models like ELMo and BERT.

Key Results

The benchmarking results highlight the difficulty of the DENS dataset and the necessity of contextual modeling. Deep learning architectures significantly outperformed traditional lexical methods, which often struggle with the ambiguity of narrative language.

Model performance comparison on the DENS dataset, with fine-tuned BERT leading at 0.604 micro-F1.

Fine-tuned BERT emerged as the top performer with a 60.4% micro-F1 score. The significant jump from traditional SVM-based models (approx. 45%) to transformer-based models underscores that identifying emotion in stories requires "attending" to themes and context that span multiple sentences, rather than relying on isolated keywords.

Discussion and Future Work

While DENS represents a significant step forward, several challenges remain. Class imbalance is a notable hurdle; categories like Disgust and Surprise were found to be too small or subjective for robust training in this iteration. Additionally, while passages are self-contained, they inevitably lose some emotional resonance when separated from the full novel's global context.

Future research directions include exploring more nuanced emotions specific to fiction—such as pain, confusion, or jealousy—and utilizing few-shot learning techniques to help models recognize underrepresented emotional classes.

Summary updated

BibTeX
@inproceedings{liu2019dens,
  author = {Chen Liu and Muhammad Osama and Anderson de Andrade},
  title = {{DENS: A Dataset for Multi-class Emotion Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1656},
}
Figure