Abstract
Emotion recognition in natural language processing has traditionally focused on short-form texts such as tweets or product reviews. However, narrative storytelling involves complex emotional arcs that require broader context. This paper introduces the Dataset for Emotions of Narrative Sequences (DENS), a new benchmark for multi-class emotion analysis in long-form fiction. DENS comprises over 9,000 passages from classic and modern literature. Benchmarking results demonstrate that while traditional lexicon-based methods struggle, fine-tuned transformer models like BERT achieve a micro-F1 score of 60.4%, highlighting the importance of contextual understanding in narrative analysis.
Introduction
Most existing emotion recognition research is constrained by the limitations of "short-form" text. While 280-character tweets are useful for sentiment analysis, they often lack the depth required to capture the nuanced emotional journeys found in literature. Narratives rely on context spanning multiple sentences to convey complex feelings. To address this gap, researchers from Wattpad developed DENS, a dataset specifically designed to challenge models to "read" more deeply into multi-sentence passages from both classic and modern storytelling.
The DENS Dataset
The DENS dataset consists of 9,710 passages with an average length of 86 words (approximately 6.24 sentences). These passages were sampled from 88 classic titles in Project Gutenberg and over 1,500 modern titles from Wattpad, covering diverse genres such as Mystery, Fantasy, and Romance.
The labeling schema is based on a modified version of Plutchik’s Wheel of Emotions. During initial surveys, the authors found that "Trust" was difficult for readers to identify consistently in romantic narratives. Consequently, "Trust" was replaced with "Love," and a "Neutral" category was added to account for non-emotive segments. The final dataset includes nine classes: Joy, Sadness, Anger, Fear, Anticipation, Surprise, Love, Disgust, and Neutral.
Methodology
The construction of DENS involved a multi-stage pipeline:
- Passage Selection: Passages (40–200 tokens) were selected using a combination of random sampling and keyword-based pruning to ensure emotional richness.
- Annotation: Amazon Mechanical Turk (MTurk) was used for labeling, with each passage reviewed by at least three annotators. Discrepancies were resolved by in-house experts.
- Benchmarking: A variety of architectures were evaluated, ranging from traditional Bag-of-Words (TF-IDF + SVM) to state-of-the-art deep learning models including Bi-directional RNNs, ELMo, and BERT.
Results and Discussion
The experimental results underscore the complexity of emotion recognition in long-form text. Deep learning models that leverage contextual embeddings significantly outperformed traditional lexical-based methods.
Fine-tuned BERT emerged as the top performer with a 60.4% micro-F1 score. In contrast, lexicon-based methods (e.g., NRC or Depeche++) performed poorly, with scores as low as 25.4%. This disparity confirms that emotion in stories is often embedded within the narrative structure and subtext rather than being explicitly signaled by individual "emotion words."
Furthermore, the inclusion of Wattpad data allowed the dataset to capture modern linguistic trends and themes—such as LGBTQ+ narratives—that are absent in 19th-century literature. However, human agreement levels (73.5% majority agreement) reflect the inherent subjectivity of interpreting emotion in art.
Conclusion
DENS provides a robust benchmark for moving emotion analysis toward more complex, narrative-driven contexts. Future work will explore multi-label classification to better capture "bittersweet" or mixed emotions and integrate common-sense reasoning to help models understand the underlying plot-driven motivations for character emotions.
Summary updated