TL;DR
This paper explores how sentiment analysis and stylistic features can predict literary quality, as measured by reader ratings on Goodreads. They found that while traditional stylistic features are helpful, incorporating sentiment-related features significantly improves prediction accuracy, suggesting that the emotional arc and sentiment dynamics of a novel play a crucial role in reader appreciation.
What Problem Does This Paper Address?
The concept of "literary quality" has long been debated, with individual tastes varying widely. However, there's an ongoing quest in quantitative literary analysis to identify textual patterns that consistently contribute to a work's appreciation by readers. While previous studies have explored stylometric features (like word choice and sentence structure) for this purpose, this paper investigates whether sentiment analysis, particularly the emotional trajectory of a narrative, can offer a more powerful predictive signal for literary quality. The challenge lies in defining and quantifying "literary quality" in a way that can be objectively measured and correlated with textual features.
Methods
The researchers employed a multi-faceted approach:
- Corpus Selection: They utilized the "Chicago corpus," a large collection of over 9,000 English-language novels published between 1880 and 2000. They also created a "filtered corpus" by removing novels with very low Goodreads ratings to assess model performance on a potentially higher-quality subset.
- Quality Measure: Goodreads average ratings were used as the proxy for literary quality. While acknowledging the limitations of a single score, they highlighted Goodreads' vast user base and diverse audience as advantages.
- Feature Extraction:
- Stylometric Features: These included measures of lexical diversity, text compressibility, readability (e.g., Flesch-Kincaid), and various syntactic features like part-of-speech frequencies and specific grammatical structures.
- Sentiment Analysis Features: They applied the VADER sentiment analysis model at the sentence level to generate sentiment scores for each sentence. From these sentence-level scores, they derived "sentiment arc features." These included simple measures like mean sentiment and standard deviation, as well as more complex dynamic features like the Hurst exponent (measuring long-term memory/persistence) and Approximate Entropy (measuring complexity/irregularity) of the sentiment arcs.
- Modeling: They used relatively simple and interpretable regression models, including Linear Regression, Lasso, Bayesian Ridge, and others, to predict Goodreads ratings based on the extracted features. The focus was on understanding the relationship between features and outcomes, prioritizing interpretability over achieving the absolute highest prediction accuracy.
Datasets
- Chicago Corpus: A collection of over 9,000 English-language novels from 1880-2000.
- Filtered Corpus: A subset of the Chicago corpus, excluding novels with Goodreads ratings below 2.5.
Key Insights and Ideas
- Sentiment Matters: The most significant finding is that sentiment-related features, particularly those describing the dynamics of the sentiment arc (like its persistence, predictability, and complexity), substantially improve the prediction of literary quality compared to stylometric features alone.
- Low-Rated Novels Have Distinct Profiles: Novels with very low ratings (the bottom 2%) exhibit a unique textual and sentiment profile that is relatively predictable. This suggests that certain stylistic and emotional patterns are strongly associated with reader dissatisfaction.
- Balance of Simplicity and Diversity: For more appreciated novels, the study points towards an equilibrium between stylistic simplicity and diversity, and between coherent emotional progression and nuanced sentiment changes.
- Interpretability is Key: The choice of simpler regression models allowed the researchers to not only predict ratings but also to understand which features were most influential, providing insights into the "why" behind reader appreciation.
Key Results
- Improved Prediction: Sentiment features consistently improved the predictive performance of most regression models across both the whole and filtered corpora.
- Best Model: Bayesian Ridge regression emerged as the best-performing model for predicting Goodreads ratings.
- Predictability of Low Ratings: The models were particularly effective at predicting ratings for novels below 2.5, indicating a discernible pattern in works that are widely disliked.
- Feature Importance: For the whole corpus, features related to punctuation, text compressibility, reading ease, and frequencies of verbs, pronouns, and adverbs were inversely related to quality. For the filtered corpus, a more sophisticated style (fewer pronouns, explicit subjects, stopwords, more nouns, and easier vocabulary) was favored. Sentiment-wise, the Hurst exponent and approximate entropy were strong predictors, suggesting that persistent, coherent, and dynamic sentiment arcs are associated with higher ratings.
Limitations and Future Work
- Limitations: The authors acknowledge that Goodreads ratings are a simplification of reader preferences and that the corpus has a geographical and cultural bias (over-representation of Anglophone authors).
- Future Work: The researchers plan to:
- Explore other quality proxies beyond Goodreads ratings to study convergences in defining quality.
- Use a larger set of features.
- Frame the problem as a classification task and explore more sophisticated models, while still prioritizing interpretability.
Summary updated