Effects of Publicity and Complexity in Reader Polarization

Yuri Bizzoni, Pascale Feldkamp, Kristoffer NielboNAACL 2025

TL;DR

This paper investigates how literary complexity influences reader reception on Goodreads, moving beyond average ratings to analyze rating distributions. They found that more complex books, particularly those deemed canonical, tend to elicit more polarized reader opinions, suggesting that textual complexity itself drives disagreement, not just the "publicity effect" of wider exposure.

What Problem Does This Paper Address?

Traditional analyses of literary success often rely on simple metrics like average ratings or sales figures. However, these metrics can mask deeper patterns of audience reception. This paper addresses the question of how the distribution of reader ratings reveals more nuanced insights into literary appreciation and disagreement. Specifically, it explores whether textual features, such as complexity, contribute to polarized opinions among readers, even when controlling for factors like popularity (i.e., the number of ratings). The authors aim to bridge the gap between traditional popularity metrics and a richer understanding of how literary characteristics shape reader engagement and polarization.

Methods

The study employs a quantitative approach, analyzing large datasets of book ratings and textual features. Key methods include:

  • Rating Distribution Analysis: Instead of just average ratings, the authors analyze the distribution of ratings using metrics like entropy (measuring the diversity of opinions) and standard deviation (measuring the spread of ratings around the average).
  • Textual Feature Extraction: A comprehensive set of linguistic features were extracted from the full texts of novels. These include measures of:
  • Readability: (e.g., New Dale-Chall formula)
    • Syntactic Complexity: (e.g., sentence length, dependency distance)
    • Lexical Richness: (e.g., type-token ratio)
    • Vocabulary Difficulty: (e.g., word entropy, perplexity)
    • Nominality: (e.g., nominal ratio)
  • Correlation Analysis: Spearman correlations were used to examine the relationships between Goodreads metrics (rating count, average rating, entropy, standard deviation) and the extracted textual features.
  • Comparative Analysis: The study compares results across a large, general dataset and a smaller, curated dataset of canonical works to understand how literary categorization influences these relationships.

Datasets

The study utilizes two primary datasets:

  • Goodreads Book Graph Dataset: A large dataset containing metadata for approximately 2 million titles, collected in 2017. This dataset was filtered to include only literary novels with at least 10 ratings.
  • Curated Corpus: A smaller dataset of 7,939 English-language novels, for which full texts were available. This corpus was compiled based on library holdings and a preference for titles with higher numbers of holdings, aiming for a more representative sample of literary works.

Key Insights and Ideas

  • Beyond the Average: The shape of a book's rating distribution (entropy and standard deviation) offers more nuanced insights into reader reception than simple average ratings.
  • The "Publicity Effect" is Real (Mostly): Books with more ratings (higher publicity) tend to have more diverse opinions (higher entropy), supporting the "publicity effect" hypothesis. This suggests wider exposure leads to a broader range of reactions.
  • Complexity Drives Polarization: Crucially, textual complexity (measured by features like perplexity, nominal ratio, and dependency distance) is strongly correlated with higher rating distribution entropy and standard deviation, especially in canonical works. This indicates that challenging texts can independently drive reader disagreement.
  • Canonical Works as a Special Case: Canonical books, while potentially having fewer readers overall, elicit more polarized responses. This suggests a "canonicity effect" that interacts with textual complexity, potentially due to heightened reader expectations or the inherent difficulty of these texts.
  • Decoupling of Popularity and Appreciation: The study found that high rating counts (popularity) do not necessarily correlate with high average ratings, suggesting these are distinct dimensions of literary success.

Key Results

  • Hypothesis H1 (Publicity Effect): Confirmed. A positive correlation was observed between rating count and rating distribution entropy in the large dataset.
  • Hypothesis H2 (Decoupling): Confirmed. A lack of correlation between average rating and rating count was observed in the large dataset.
  • Hypothesis H3 (Textual Complexity): Partially confirmed. Several markers of literary complexity showed positive correlations with rating distribution entropy, particularly in the canon subset.
  • Perplexity was the strongest predictor of rating polarization (entropy/SD).
  • Nominal ratio and dependency distance also showed stronger correlations with rating variability in canonical works.
  • Canon vs. Full Corpus: Correlations between textual features and rating distribution metrics tended to be stronger in the canonical subset, suggesting that intrinsic textual qualities have a more pronounced effect on polarization within this group.

Limitations and Future Work

  • Dataset Bias: The study is primarily focused on English-language novels, with a potential bias towards works by male authors. This limits the generalizability to other genres, languages, and author demographics.
  • Canonicity Definition: The binary classification of "canon" vs. "non-canon" might oversimplify a more continuous concept, and a more nuanced approach could yield further insights.
  • Goodreads User Base: Goodreads' user base is predominantly anglophone, which may not represent global reader sentiment.
  • Future Research: The authors suggest expanding the analysis to include other genres, datasets, and longitudinal studies to track how ratings evolve. Incorporating reader demographic information and exploring extra-textual factors like author and reviewer gender are also proposed avenues for future work.

Summary updated

BibTeX
@inproceedings{bizzoni2025effects,
  author = {Yuri Bizzoni and Pascale Feldkamp and Kristoffer Nielbo},
  title = {{Effects of Publicity and Complexity in Reader Polarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure