DENS: A Dataset for Multi-class Emotion Analysis

Chen Liu, Muhammad Osama, Anderson de Andrade

Abstract

Emotion recognition in natural language processing has traditionally focused on short-form texts such as tweets or product reviews. However, narrative storytelling involves complex emotional arcs that require broader context. This paper introduces the Dataset for Emotions of Narrative Sequences (DENS), a new benchmark for multi-class emotion analysis in long-form fiction. DENS comprises over 9,000 passages from classic and modern literature. Benchmarking results demonstrate that while traditional lexicon-based methods struggle, fine-tuned transformer models like BERT achieve a micro-F1 score of 60.4%, highlighting the importance of contextual understanding in narrative analysis.

Introduction

Most existing emotion recognition research is constrained by the limitations of "short-form" text. While 280-character tweets are useful for sentiment analysis, they often lack the depth required to capture the nuanced emotional journeys found in literature. Narratives rely on context spanning multiple sentences to convey complex feelings. To address this gap, researchers from Wattpad developed DENS, a dataset specifically designed to challenge models to "read" more deeply into multi-sentence passages from both classic and modern storytelling.

The DENS Dataset

The DENS dataset consists of 9,710 passages with an average length of 86 words (approximately 6.24 sentences). These passages were sampled from 88 classic titles in Project Gutenberg and over 1,500 modern titles from Wattpad, covering diverse genres such as Mystery, Fantasy, and Romance.

Table 1: Examples of narrative passages and their corresponding emotion labels in the DENS dataset.

The labeling schema is based on a modified version of Plutchik’s Wheel of Emotions. During initial surveys, the authors found that "Trust" was difficult for readers to identify consistently in romantic narratives. Consequently, "Trust" was replaced with "Love," and a "Neutral" category was added to account for non-emotive segments. The final dataset includes nine classes: Joy, Sadness, Anger, Fear, Anticipation, Surprise, Love, Disgust, and Neutral.

Methodology

The construction of DENS involved a multi-stage pipeline:

  1. Passage Selection: Passages (40–200 tokens) were selected using a combination of random sampling and keyword-based pruning to ensure emotional richness.
  2. Annotation: Amazon Mechanical Turk (MTurk) was used for labeling, with each passage reviewed by at least three annotators. Discrepancies were resolved by in-house experts.
  3. Benchmarking: A variety of architectures were evaluated, ranging from traditional Bag-of-Words (TF-IDF + SVM) to state-of-the-art deep learning models including Bi-directional RNNs, ELMo, and BERT.

Results and Discussion

The experimental results underscore the complexity of emotion recognition in long-form text. Deep learning models that leverage contextual embeddings significantly outperformed traditional lexical-based methods.

Table 2: Performance comparison of various NLP models on the DENS dataset using micro-F1 scores.

Fine-tuned BERT emerged as the top performer with a 60.4% micro-F1 score. In contrast, lexicon-based methods (e.g., NRC or Depeche++) performed poorly, with scores as low as 25.4%. This disparity confirms that emotion in stories is often embedded within the narrative structure and subtext rather than being explicitly signaled by individual "emotion words."

Furthermore, the inclusion of Wattpad data allowed the dataset to capture modern linguistic trends and themes—such as LGBTQ+ narratives—that are absent in 19th-century literature. However, human agreement levels (73.5% majority agreement) reflect the inherent subjectivity of interpreting emotion in art.

Conclusion

DENS provides a robust benchmark for moving emotion analysis toward more complex, narrative-driven contexts. Future work will explore multi-label classification to better capture "bittersweet" or mixed emotions and integrate common-sense reasoning to help models understand the underlying plot-driven motivations for character emotions.

Summary updated

Abstract

We introduce a new dataset for multi-class emotion analysis from long-form narratives in English. The Dataset for Emotions of Narrative Sequences (DENS) was collected from both classic literature available on Project Gutenberg and modern online narratives avail- able on Wattpad, annotated using Amazon Mechanical Turk. A number of statistics and baseline benchmarks are provided for the dataset. Of the tested techniques, we find that the fine-tuning of a pre-trained BERT model achieves the best results, with an average micro-F1 score of 60.4%. Our results show that the dataset provides a novel opportunity in emotion analysis that requires moving beyond existing sentence-level techniques.
BibTeX
@inproceedings{liu2019dens,
  author = {Chen Liu and Muhammad Osama and Anderson de Andrade},
  title = {{DENS: A Dataset for Multi-class Emotion Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1656},
}
Figure