Modeling Readers’ Appreciation of Literary Narratives Through Sentiment Arcs and Semantic Profiles

Pascale Moreira, Yuri Bizzoni, Kristoffer Nielbo, Ida Marie Lassen, Mads ThomsenACL 2023

TL;DR

This paper explores how to predict readers' appreciation of literary narratives by combining sentiment analysis of a story's emotional arc with semantic profiling of its content. They found that this combined approach significantly improves prediction accuracy compared to using either method alone, suggesting that both the emotional journey and the thematic elements of a story are crucial for reader engagement.

What Problem Does This Paper Address?

Quantifying and predicting "literary quality" is a long-standing challenge in computational literary studies. Defining quality itself is complex, as it's influenced by a vast array of text-intrinsic features (like style and narrative structure) and text-extrinsic factors (like author reputation or reader demographics). While some studies have shown that text features do impact perceived quality, it's still unclear which features are most important and how they interact. This paper aims to tackle this by investigating the predictive power of a narrative's emotional trajectory (sentiment arc) and its thematic content (semantic profile) on reader appreciation, using book ratings as a proxy for quality.

Methods

The authors employ a stacked ensemble machine learning model. This involves:

  • Sentiment Features: They use VADER (Valence Aware Dictionary and sEntiment Reasoner) to extract sentiment scores for each sentence in a novel. This allows them to construct a "sentiment arc" representing the emotional progression of the story. They also derive dynamic features from these arcs:
  • Hurst Exponent: Measures the persistence or long-range dependence of the sentiment arc, indicating trend-following or mean-reverting behavior.
  • Approximate Entropy: Measures the predictability and complexity of fluctuations in the sentiment arc.
  • Semantic Features: They utilize Roget's Thesaurus to create semantic profiles of novels. By mapping words in a text to Roget's categories, they can represent the distribution of themes and topics within a narrative.
  • Stacked Ensemble Model: A Support Vector Machine (SVR) and a Random Forest regressor are used as base models, with their predictions fed into a Ridge regressor as a meta-classifier. This ensemble approach aims to leverage the strengths of multiple models.
  • Feature Selection: A filter method is used to rank features by relevance and select the most informative ones to reduce model complexity and improve interpretability.

Datasets

  • Chicago Corpus: A dataset of over 9,000 English-language novels published or translated between 1880 and 2000, selected based on library holdings. This corpus aims to be diverse, including popular fiction and highly esteemed works.
  • GoodReads Ratings: Used as the proxy for reader appreciation. The authors acknowledge limitations of this metric, such as its potential conflation of genre-specific preferences and its mono-dimensional nature, but highlight its accessibility and the large number of users providing diverse ratings.

Key Insights and Ideas

  • Sentiment Arcs Matter: The emotional journey of a narrative, not just its overall sentiment, is a significant predictor of reader appreciation. The dynamics of sentiment (persistence and predictability) are particularly important.
  • Thematic Content is Crucial: Semantic profiling using Roget's categories provides valuable insights into the thematic elements that contribute to a novel's perceived quality.
  • Synergy of Features: Combining sentiment and semantic features significantly outperforms models using either feature set alone, suggesting that both the "what" (themes) and the "how" (emotional journey) of a story are critical.
  • Predictability of Low-Rated Titles: Surprisingly, the worst-rated titles in the dataset exhibited a marked predictability. This might be due to them having more distinct or easily identifiable sentiment/topical profiles, which the model can latch onto, potentially inflating overall performance metrics.
  • More Data = Better Predictions: Increasing the number of reader ratings for a book consistently improves model performance, suggesting a shared perception of literary quality among readers and hinting at objective criteria.

Key Results

  • The combined sentiment and Roget features achieved the best performance, with an R² of 0.24 and an MSE of 0.04 on the full dataset (9089 titles).
  • Using only sentiment features yielded an R² of 0.07 and MSE of 0.06.
  • Using only Roget features yielded an R² of 0.23 and MSE of 0.05.
  • The baseline model (average sentiment) performed poorly, with a negative R² (-1.1) and an MSE of 0.8, highlighting the inadequacy of simplistic sentiment analysis.
  • Models trained on texts with a higher number of ratings (e.g., >130) consistently showed improved performance, indicating greater reliability in reader scores.

Limitations and Future Work

  • Stylometric Features Omitted: The model does not incorporate traditional stylometric features (word choice, sentence structure, punctuation), which are crucial for capturing "literariness" and stylistic nuances.
  • GoodReads Bias: Reliance on GoodReads scores introduces biases related to genre preferences and reader demographics.
  • Limited Corpus: The study is limited to English-language texts from the 19th and 20th centuries, potentially limiting generalizability to other periods, languages, or contexts, and excluding the impact of translation.
  • Future Work: The authors plan to:
  • Compile larger datasets with more features, including stylometric and syntactic features.
  • Investigate genre-specific patterns to reveal unique relationships within different literary types.
  • Explore alternative quality metrics beyond GoodReads, such as anthologies, awards, and canon lists, to gain a more nuanced understanding of literary quality.

Summary updated

BibTeX
@inproceedings{moreira2023modeling,
  author = {Pascale Moreira and Yuri Bizzoni and Kristoffer Nielbo and Ida Marie Lassen and Mads Thomsen},
  title = {{Modeling Readers’ Appreciation of Literary Narratives Through Sentiment Arcs and Semantic Profiles}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure